Letter to my successors (or what do I know about the Xorg bug triage)

Út 22 listopadu 2011

For various reasons we have agreed with my supervisor, that my working on the Xorg bug triage as a full time job will be terminated (feels like a relief) and I will switch to working as a programmer for the Red Hat internal Desktop QA team (which is a good thing!). As the last act in my old position I was asked to write down brain dump of my experience working on the Xorg bug triage for my potential successors. I am trying also to pay some debts I feel I have in documenting some parts of my bug triage workflow and tools.

First of all, it is not like there isn't any information available already. We as a BugZappers team tried to collect as much information as possible so anybody who would like to join our ranks would have something to chew upon without us repeating it again and again. So, there is How to Triage page and other pages coming from this navigation page (e.g., you are expected to know bug workflow by heart if drawn in the midnight from your bad).

Queries

Probably the most important thing which can help you to survive a terror of the never-ending inflow of new bugs are well prepared queries. Red Hat Bugzilla (or any other bugzilla for that matter, as far as I know), doesn't have a really good way to create coarser (or finer) organization of bugs than on the level of components and products. The situation is especially bad, because products are occupied by, well, products (like Fedora, RHEL, etc.), so in fact the only classification of bugs we have is on the basis of components. Which is obviously crazy huge group of bugs for some components (kernel) and too little for many others. Other options (using keywords in whiteboards, for example) requires much manual additional work (although we use it as well, see below for Xorg bugs categories).

The only way I found how to make dissecting bugs in useful ways possible is by using queries. It is normal state of things for me to have more than 50 defined queries in the bottom of my bugzilla pages. As an example (and possibly foundation of your further development) I could present you this search. Notice that it selects just some components of bugs (using a regular expression), eliminates bugs with needinfo flag set (which is thus a very important variable to care for), and eliminates already Triaged bugs. Of course, this is just a basic search which can be further narrowed by selecting only bugs in some states (although bug triagers deal mostly with NEW bugs), bugs changed in the recent time (when the time is short, it is probably better to deal with the active bugs, not stale ones), bugs for some versions of products, etc.

Firefox addon

Although there has been concerned spent lately on making bugzilla work better for developers, it is still true that its user interface is (if for anybody) optimized for users and so there is little help for avoiding repetitive work. In order to make my work with bugzilla more palatable, I have created a Firefox addon which provides tools for mechanizing a lot of stereotypic work. For example, reporters in the Red Hat bugzilla have no meaningful guidance when filing Xorg bugs, so many filed bugs are lacking logs and other information which is required for developers to effectively diagnose the issue. Actually, making sure that all this information is provided is the main task of Xorg bug triage (where normal other tasks of bug triage, e.g., deduplication and testing reproducers is hardly possible or completely impossible given the limits on available hardware and knowledge of the code itself).

Back to the firefox addon. The main functionality has been described on this blog, so just in the way of summary. The main point which you need to understand is that almost all functionality can be controlled and substantially changed by editing a JSON file which is loaded on the startup of the addon. Per default it is this JSON, but clicking on “Triage configuration” link you can enter URL of any JSON file anywhere on the Internet. All possible options are switched on in my JSON, which can be used as an example for developing of your own file. Let me go through the main options available in such file.

Most famous and probably still most used are buttons filling standardized comments. That's the biggest parts of JSON file in the object (using Javascript terminology) commentPackages. The name evokes that there are multiple packages which can be switched on for particular bugzillas (one bug triager can certainly work on multiple bugzilla instances ... see configData.enabledPackages for examples how to switch on some packages for particular bugzillas). Each packages contains multiple objects one for each button which looks like this:

"addnoresponse": {
           "name": "NoRespns",
           "position": "topRow",
           "commentIdx": "noResponseString",
           "markTriaged": true,
           "status": "CLOSED",
           "resolution": "INSUFFICIENT_DATA"
      },

Each object has its own one-word name, which serves as an index and id for the button itself. Each object is best understood as a package of commands which will be executed when the button is pressed. Some of these “commands” actually don’t are not commands but metadata ("name" contains the label on the button, "position" shows to the addon where the button should be located), but most of them actually do something. So in this example, “commentIdx” points to the comment string in “commentString” section of the JSON file. For shorter comment texts, button package can contain directly “comment” property with the text itself. Either of these texts can be included in the comment. “markTriaged” will obviously make all necessary steps to make bug triaged, and the remaining two commands will close the bug. “position” deserves a little bit special treatment. It can be either “topRow” or “bottomRow” which means that the button will be inserted into the row of buttons above or below the comment box respectively. “position” property can also contain the ID of the element after or before which the button should be included (such string starts with “+” or “-” before the ID string). So for example, “addexternalbugid” package contains "position": "+external_bugs_headline", which means that the button should follow HTML element with “external_bugs_headline” ID.

After “commentPackages” and “commentStrings” sections, the next one is “configData” which is a collection of various configuration options. I have in my JSON file almost all of possible ones on. I will return to this configuration section in a minute, but now let me just the mention the last section “constantData” which contains various constants for Xorg bug triagers, and which you should probably just take for your JSON.

Back to the “configData” section. The biggest object (at least in my file) is “killNodes” which is an object which for each bugzilla instance where the triager is supposed to work lists IDs of all elements you would like to remove from the page (some people especially from Mozilla complained that there is just too much stuff on every page, so that for some people the bugzilla is almost useless, but I had to admit I used it well even with bugzilla.redhat.com). “downloadJSON” contains URLs for additional data files which can be loaded for data which are just too big to be included into “constantData” section (take a look at that file). “suspiciousComponents” is for situations when reporters continuously file bugs into components which are there just for legacy or organizational reasons (xorg-x11 and xorg-x11-drivers). The last is set of couple of Booleans, which directly allow some features of the addon, which are usually not useful for “normal” bug triagers, but were useful for me:

parseAbrtBacktraces
This is mostly legacy option, for large piece of functionality, which pulled just a important part of backtrace from the attached abrt-produced log.
submitsLogging
This is a switch for a functionality which is completely useless for anybody who is not hired for being a bug triager. This addons could ask you for every click on the Submit button to provide short message, which could be then collated into something which goes to my weekly status report. Obviously it should be set to false for almost everybody.
verboseInlineHistory
This is a switch for unfinished attempt to merge in functionality from Bugzilla Tweaks functionality of which was later included into the upstream bugzilla itself.
XorgLogAnalysis
This is a tool which could be very useful for any Xorg bug triager. However, it is kind of dangerous, because if misused it could lead to quite sloppy work. What it does is that it adds to every Xorg.0.log (and dmesg output) a button which makes the addon to run the log through regular expressions which selects lines which can be interesting. Obviously it is useful for fast overview of what's going on, but certainly detailed inspection of the log is required for any decision.

So, this is finally written down documentation for the mythical JSON file, which I promised for so long.

A bit on process of treating a bug

I will skip technical issues (they are described reasonably well on the above mentioned BugZappers pages), but let me add here couple of notes on the background of the process. First of all, although bug triage is kind of lonely job, we stand on shoulders of some giants (or at least we could if we knew about them). Luis Villa (former bugmaster of Ximian and Gnome project) has a nice collection of his presentations and talks. Highly recommended. Also a lot of discussion about bugs management (obviously) happens in the Bugzilla community, for which Bugzilla Planet is a great entry point (or you can go all the way to Mozilla Planet)

(... to be continued ...)

Category: computer Tagged: bugTriage firefox jetpack