Hello! This is the post about my week 12 and, probably, also my final post for GSoC 2020.

You can see my Merge Request in regards to the support of text annotation here. Today I will talk about what I have done this week and also show the progress of marK so far near the end of GSoC.

I have implemented and fixed what I mentioned in the previous post and there are more commits that I want to highlight:

• Do not allow edit an existing polygon if is dragging

A small change (only two lines) that prevents a bug that can happen when the user is annotating data with a mouse click and move and the current polygon pass over an existing polygon, now it is not allowed to do so.

• Add support for mouse double click event in text painter

This should make the life of the users easier if they need to annotate a single word per entity, with the speed of only a double click, also you can achieve the same behavior (select a whole word) clicking in a sentence that has only one letter.

• Use eventFilter to treat mouse events in the container instead of overridden functions

Previously the container of annotation (aka Container) overridden functions to treat mouse events like mousePress, mouseMove, mouseRelease, and (recently) mouseDoubleClick, now it uses an eventFilter that treat those mouse events. This implementation will handle mouse events in any Widget as long as this eventFilter is installed on it (or its viewport) making it work pretty well with the painters structure.

• Use enum to treat the change of Painters

This one is an improvement of a previous commit, Create a new painter instance just when needed, now marK has a more robust way to treat the changes of the working painter, and it will be easier to add and modify the buttons on the left of the Container labeled as “Options”, this is nice to have as marK in the future will support other types of annotation (such as audio and video) and those may need specific UI items.

• Use KXmlGuiWindow instead of QMainWindow

Now marK shares more similarities with other KDE projects using KF5 (more specifically KXmlGuiWindow). This gives to marK some of the hackability found in them (e.g. now you can change the default shortcuts if you want) and it is also easy to have a pretty help and settings menu.

Now, I had like to show some features of marK that made into the main branch during this GSoC period, or will have after the MR is merged:

Summarizing:

• use temporary files to load annotation made before switching to another item (made into the main branch)
• autosave functionality (made into the main branch)
• support to mouse click and move to annotate data (and double click for text annotation)
• painters (see week 4 and 5, if you do not remember about it)
• use of KDE Frameworks 5
• support to text annotation

There is still some work that needs to be done but I am already happy to show this development (and I will be even happier after the code is merged : ).

Thanks

In these months of GSoC, I have received help from Tomaz Canabrava, Sandro Andrade, and, the one that helped and guided me the most, my mentor Caio Jordão Carvalho, I am very grateful and thankful for it. I had like to thank you all for the attention and opportunity, I hope that I didn’t let you down. 🙂

What is next after GSoC? I am already talking with Caio and we are planning some cool things to make marK better, expect more posts and news 😉

That is it, see you in the next post 😉

Hello!

In these three weeks, I focused on general bug fixes (majority caused by me btw) and also on improving the logic behind text annotation:

• Show current directory in listLabel containing

A minor fix, now marK shows the initial working directory, the one where it was open at, in the listLabel (that is above the list of items).

• Reset containerWidget properly when changing directory

When changing the working directory and the current painter was the textPainter, the TextEdit instance would not be closed properly, thus not behaving as it should, as the imagePainter is the default in the containerWidget (widget where data annotation is made) when there is no item opened.

• Draw properly a rectangle in mouse move event

Like I have said in the previous post, I have added support for the mouse move event when annotating data. With this, a bug was introduced when mouse click and move happened in the rectangle mode (a bunch of mini rectangles would be created instead of just adjusting one, as in the example shown above).

• Fix scaling problem of polygons

With the new logic of the painters, this bug was introduced again, in the past, marK had this problem with images bigger than 1280X960 (that I solved in SoK 2020). To solve this, I used a second vector that contains temporary objects adapted to the containerWidget, and the main vector, which also contains annotation data, will have the correct values meant to be saved. This means that image annotation is now working properly again.

• Better handling of text annotation

I have improved the logic of text annotation, now the code is more maintainable and works like the imagePainter, creating new MarkedObjects (the class that represents annotation data) just when it is needed, saving the annotation data just when it is understood that it is already complete (mouse release), thus avoiding some saving (like always triggering the autosave functionality as before) and deleting operations that may affect the performance. A bit later I improved it further, fixing one case where the end of current text selection is at the beginning of another causing the latter to disappear and improved the readability of the code.

What is next? There is still one part of the code that I trying to improve in regards to text annotation, currently, it is not possible to use the wheel to navigate through a document that is big enough to not fit in the whole containerWidget, this is a bug that I want to solve, I also will improve (and add) options to the menu of marK to its first release.

That is it, see you in the next post 😉

Hi! Today, I am bringing the results of my work in developing text annotation support for marK.

In these three weeks I have been working in bug fixes and the logic to handle text annotation. It is worth mentioning one change in the original plan, I said in the previous posts that I was going to use KTextEdit, but for now, the code is using QTextEdit that is already available by using Qt and I could focus on the logic.

In the first day of the coding period I made a post that explained a bit about text annotation, now here are two examples that are already possible to do in marK:

Named Entity Recognition

Phrase Chunking

Example of current usability

Above is a short video that I made to show the current state of text annotation in marK, near to the end, you can also see the file generated that contains the annotation data (a JSON file in this case).

This coding period was quite challenging, but also fun, I have learned a lot as I was implementing the logic to handle text annotation. Also, as you could see in the video, now it is possible to annotate data with mouse click and drag instead of only clicking, I think it will improve the user experience, making marK more comfortable to use, and it is not only limited to text annotation, image annotation supports it too.

What is next? I will fix the bugs that I found, improve the UX, make the code more performant and, together with my mentor, prepare marK to have his first release.

that is it, see you in the next post ; )

Hi, today I will talk about my week 4 and week 5 and bring some news!

The last post was short but this one will make up for it, explaining some important bits, and changes, in the structure of mark that changed/improved during the first month of coding in GSoC.

In week 4, I documented a huge part of the existing code, although there is still a need for some updates. Currently in week 5, I am fixing some bugs of the new logic and I will document the newly created Painter class (more information below), also start developing the logic for text annotation.

The structure of marK had some changes that I had like to highlight and explain. this diagram represents the current structure, with marK also being the main window class.

Container was an abstract class and its children handled the loading of the necessary items, such as the image for the previous ImageContainer and text for the to be TextContainer, and also annotate the data. After talking with my mentor, we decided that it could change for better and be simpler.

Now marK has only one Container, this one is a “canvas” to the Painter and he will be switched accordingly to the file type, the painters now are the ones responsible for load and display the contents of the files, and also handle the annotation of data.

The container holds the current MarkedObject being annotated and also a vector of the previous ones. With this change the code is smaller and it is easier and faster to change between different types of annotation.

MarkedObject represents the annotated data. Each MarkedObject has a reference of a MarkedClass, and use it to have a color and identifier for the annotation. It also uses d pointer to avoid problems related to ABI changes (as I said in the previous post).

The existing markedClasses are shown in the marK’s comboBox, being possible to select and also modify the name identifier and color. Allowing the user to personalize and edit accordingly to his needs.

Serializer is responsible for reading and saving the MarkedObjects resulted from the annotation in marK. Its functions have been refactored to support text annotation (and others types of annotation). Currently, serializer only exports to two file formats, xml and json.

Painter is the base class of all Painters and is friend of the Container, as said before, it takes over the responsibility of the containers (of the previous logic). There will be a derived class for each type of annotation (such as the ones shown above). Currently, only ImagePainter exists and works but this month I will develop the TextPainter and change this.

In the next post I will explain about the TextPainter and show the initial text annotation in marK.

That is it, see you in the next post ; )

Hi! This is my report of the weeks 1, 2 and 3 of GSoC 2020.

First of all, sorry for taking a while to write the first post of the coding period, before writing I was making sure that everything was working properly and that I hadn’t broken anything, well now lets go to the actual report.

I have fixed the build errors of marK and merged the code of the branch of SoK 2020 that had yet to reach master, !2 . Also I started the implementation of text annotation.

Unfortunately I have nothing visual to show has the modifications do not change anything GUI-related, but there are things worth mentioning:

• Use of opaque pointers, this is an important step for plugins support in the future.
• Migrated the image annotation to its rightful place and separated it from the core of marK.
• Improved the logic of the class that write/read annotation data to/from json and xml files.

That is it, see you in the next post ; )

Hello! As I said in the previous post I will be posting in this blog about my experiences in GSoC 2020 (if you do not know about it, see my first post).

Community bonding period has ended and officially the coding period begins now. This is my second (and late) post and I will talk about one of my main objectives in this project, text annotation, but first a little introduction:

In a supervised learning stage, data annotation is indispensable to machine learning models, so it can learn to recognize predetermined patterns and the algorithm can treat new, non-annotated data and successfully do its task. marK is a machine learning dataset annotation tool that aims to facilitate the important process of annotating data.

Text annotation

Text annotation, one type of data annotation, is the task of labeling text-based data. The acquired metadata make possible to train the learning model to recognize patterns to tackle a huge set of problems and niches. It has a bunch of fields, each one meant to a specific niche/objective, such as:

Phrase chunking

Phrase chunking consists of labelling parts of the text according to their grammatical meaning such as noun, verb, adjective, adverb and prepositional phrase, abbreviated as NP, VP, ADJP, ADVP and PP, respectively.

Named entity recognition

Named entity recognition (NER) represents a named entity in the text, these entities are labelled with predetermined labels such as corporation, localization, person, etc. Used to discern and recognize selected entities in a text.

Named entity linking

Named entity linking (NEL) is used along side with named entity recognition, its task is to link entity mentions to a corresponding entity in a external knowledge database such as Wikipedia.

By no means this was an exhaustive list, it is meant to list some possibilities of text annotation.

How text annotation should be like in marK

As of how the graphical interface should become I am still not sure, while I studied tools of text annotation for machine learning I perceived that it has a lot of potential to be better than I previously thought. Text annotation in marK should be as flexible as possible allowing the user to annotate easily and comfortably, for this I will talk with my mentor Caio and figure it out what could be the most reasonable way of doing it.

Behind the GUI, marK will have a whole subset of classes that will that care of tasks related to text annotation, having a bridge to the API KTextEditor that will play a big role in this part, being the one responsible for displaying the text and allowing its selection. marK also is going to have classes that will represent the metadata acquired in the annotation, holding the information and afterwards it will be used to generate the output (currently a JSON or XML file).

With this I hope that I have clarified and explained a little better about one of my main goals in this project.

That is it, see you in the next post 😉

Hi! Today, I am bringing some good news. The Google Summer of Code 2020 results were announced and I was accepted as a student!

I am excited and grateful for this opportunity that KDE community has given to me and I will focus to do an excellent work during this project. 🙂

I  will be working on marK, a machine learning dataset annotation tool, which I have already contributed during Season of KDE 2020. If you don’t know about it, please check my status report.

And here is a brief description about what I am going to do during this program and an explanation about some of my plans to accomplish all the objectives:

Improving marK codebase

I will improve the codebase of mark to make it extensible, making easier to add new types of annotation, e.g. text and audio annotation. To accomplish that, I will separate the image annotation logic from the current codebase, and improve wherever possible. The new core of marK will take care of different tasks related to annotation of multiple types of data.

Implementing text annotation support

Sketchy idea of how text annotation may be in marK

First, I will explain a bit about text annotation, which is the task of labeling text-based data. It involves the process of highlighting and tagging the desired terms in a document or text and its result can be used to train machine learning models for different purposes, e.g. entity linking and text classification.

For now, marK only supports image annotation. After finishing the aforementioned objective, I will add support to text annotation, using some Qt and KF5 structures for text manipulation, such as KTextEditor. These APIs are going to be helpful as I will integrate them with new components that will handle tasks related to text annotation such as labelling.

To provide a visualization of how the annotated output will be, here is how I am planning to serialize its JSON, which will be similar to the format that is current being used for image annotation:

Conclusion

It is worth mentioning that I will take advantage of the first phase of GSoC to study more about how text annotation works and improve my knowledge about Qt, software engineering (more specifically how to write good, maintainable code)  and, of course, bond with the community.

My GSoC experiences and progress will be published in this blog, also my proposal can be found here.

That is it, see you in the next post 😉