Testing and Feedback in Ubuntu
I was reading an article about how Ubuntu is a bad standards barer for the “Linux” desktop. I’ll leave aside how paradoxical the brand “Linux” is used to mean desktop when it means nothing of the sort and I assume she means FreeDesktop (FDO).
Now I know that she’s biased and makes a few mistakes, for instance inferring that fedora is made by Red Hat alone, claiming that Red Hat has any sort of desktop strategy at all other than “for geeks”. But don’t let her red hattidness mean that there isn’t a good point about testing.
But I was struck by the problems that she has had and the comments to the entry. When comparing them to my own support roster for the past few months of sudden grub mortality (8 cases) where grub just looses all ability to boot anything with cryptic errors such as “Invalid symbol ‘u’ found” and ‘Error 15′.
Other problems concerning hardware or software failures are popping up more too. It’s actually quite worrying the amount of hand holding I’m having to do for new installs of Ubuntu. Not just the usual restricted drivers, restricted extras, libdvdcss and sun-java installation, but more messing about to get systems stable.
So, what do I think we should do about some of these problems? Testing is one way of getting stuff working, but our testing procedure for releases isn’t organised properly. We simply release a candidate and let users use it until it breaks, very unstructured. As a general testing that’s fine, but I think we should be asking users to sign up specifically to test specific things. Hardware they have, software they use, and that this process could well be made much easier by having more automated testing that executes greater depth of features and mimics facets of use better. If that’s not possible then having better defined instructions of how to test your hardware and having ways of inputting that into reports that are just as easy to fill out.
Lists of hardware and software, who has tested them, what the results are, what isn’t being tested (which requires us to know of every piece of hardware too). We need users who test to feel important in what they test and the reports can help us pick out issues, cut down on the bureaucracy of bug handlers between the users and the developers that simply invalidate bugs because there is a new version without even testing to see if it’s fixed. We can see what is being tested a hundred times and what never gets tested, we need to furnish developers with a way to collect together a list of testers to form an informal community to test new versions of things for patch fixes.
I think we’re on the way to making beta/alpha releases easy to install as secondary systems and packages already allow us to move between versions of a package. More work is probably needed to make it all fluid and easier. It would be nice if packages in ubuntu had live tests that exercised their installed state, something that could be run on any chipset and reported back.
I really do think we need to do far more that what we’re currently doing, and not just more testers, but more structures for QA to grow upon. But of course this requires resources, we have a community willing to do stuff but perhaps we should have some paid QA Community people who can provide the structure and really think in depth about how to exercise the community’s enthusiasm and self interest in testing new releases.
Your thoughts?
Update: People have commented quite rightly that we have a major deficit in programming time. I attribute this to lack of economic viability, others are attributing it to lack of skill in the willing participatory community. But for now I just want to talk about making testing easier and the results more useful and directly targeted at developers.
Tags: Feedback, QA, Testing
If you haven’t seen this before, it’s an eye-opener:
http://video.google.com/videoplay?docid=-3942663057311698508
Aside from more substantive issues with actual software, I think if the Lucid design changes landed with this kind of information backing them up, there wouldn’t be the furor we’ve seen.
“Testing is one way of getting stuff working” – No. Testing is a way of finding stuff that’s not working. Getting stuff working requires people fixing stuff. I agree with your points about better testing, but we already find far more problems that we can fix. Better testing is great, but I think it’s secondary to getting more people involved in fixing if we want improvements.
How about adding upstream (at gnome.org) an app called gnome-functionality-checklist that will auto-test as much as possible and guide users through confirming the automatic tests and then testing all the functionality that can’t be auto-tested.
I know in my very limited experience submitting bugs I have always had the same result of no fix. I also recall almost all responses to the bugs I’ve submitted being other users confirming and waiting weeks/months before even being acknowledged by a developer. I may be a bad example but for me to become enthusiastic better feedback and better explanations would have to be provided.
I can understand something not being fixable for technical or economical reasons and if that is the case they should say such rather than classifying it as a wish-list item and dismissing, etc.
P.S. I am very grateful for Linux and for those who work so hard to make it.
That’s an economic issue which few people are willing to admit. There seems to be an odd perception that if we can just get more developers involved… perhaps more developers would be involved if they didn’t have to be working full time jobs stacking boxes at a supermarket. Wouldn’t it be nice if the people needing the fix could somehow pay the developers to fix it?
No serious efforts have been made to remedy that part of the resource problem.
@doctormo(2010-04-11 at 02:38)
It is certainly a complicated task. How do you guarantee the work? How do you raise enough to make the developers time worthwhile? Should it be fixed upstream or not and who should get the money or what portion of it, e.g. writing ALSA gets no payment but fixing a bug in the default config deserves reward?
I have high hopes for http://flattr.com/ but that is really for content creation. Some similar idea might work for open software (dividing a subscription fee among a user’s most used apps).
I concur with Nathan and have had the same experience. Nowadays I do not report bugs because it merely seems to be a way of wasting my time. I switched to Lucid two weeks ago and have encountered several issues like that in the original article. Since I pay $0 for Ubuntu I just live with them.
The main focus should be on bringing more developers to Ubuntu. The organization of the work-flow is also very important but in my case I have reported more than 50 bugs so far and my reports concern Jaunty, Karmic and Lucid. 2/3 of them have been confirmed and since then no fixes or any comments have been submitted.
The speed of fixing new bugs is not good enough! And Ubuntu should take care a lot more for the upstream bugs. This DOES NOT happen.
I have a very fresh example. There was a really serious issue in OpenOffice and the problem was connected only with the go-oo (and not the official spin of OpenOffice). The bug report in Ubuntu was maybe 2 or 3 years old… then I reported the bug in Novell bugzilla and the bug was resolved after 2 months. Of course, Ubuntu developers said “Fix released”… and actually did not mention that the fix was not committed by them.
I agree to the “testing is not fixing” point of view… but I think improving testing abilities in ubuntu IS essential especialy we should make easier for developpers to add custom tests or survey for ubuntu users through the desktop.
Mozilla Labs have the “test pilot” extension (http://testpilot.mozillalabs.com) that is incredibly well done and designed… and they wish to improve/extend it by allowing users to beta test new firefox feature in one clic (http://azarask.in/blog/post/letting-firefox-move-faster-solving-the-innovators-dilemma/)… we really should get inspired by those practices. No?
Nathan: Yes, I’ve watched closely and there are a few things that are worth thinking about:
a) A developer fixing the bug because he is effected by it, is investment
b) Users must be culturally expected to invest in improvements
c) If there is not enough money, then the bug can be considered “uneconomical”, since no one is willing or able to pay for it’s fix.
This free as in beer culture is poisonous IMO.
Well, I usually try not to attack people when disagree with them and attack the problem instead, but still when I read the article of this lady and her comments I have the feeling, that it is a bit astroturfing for RedHat – I mean she said something like “how dare they compare ubuntu with big distros/companies with lots of developers behind them like Red Hat and Novell” and adding the fact the she is/was Red hat employee and adding the fact that she said – I will do bug fixing for ubuntu if the give me a good salary I really think that she attacks ubuntu/canonical for some personal beliefs/viewpoints/problems, and not because ubuntu has problems (which of course it has). So dont give her much credibility for what she is saying. That does not mean she is not partially right though – ubuntu has some very old bugs not fixed, but if it is so much dominated in every possible aspect than other distros why is it popular? Other companies, have more money, more developers, much bigger advertisement budget, bigger clients, and still ubuntu is synonim to linux for the average user. So for all the people helping ubuntu getting better (affiliated with canonical or not) – keep up the good work, some people are frustrated that their employers dont have the popularity ubuntu has and probably are angry of the bargain (money the had invested to popularity they have).
P.S. When writing in english my posts always sound childish, dont know why
What’s this about installing Java? I thought Java was open source now and could be packaged with Ubuntu?
Red Hat does have a desktop strategy for non-geeks. Why else would they invest in all the core pieces of the Linux desktop, everything from NetworkManager to GNOME Shell? Perhaps you are jumping to conclusions without the support of data as well.
Rahul: That’s not strictly true, Red Hat have enterprise desktop aspirations and they do need desktops internally. Plus staff seem to be able to choose projects half the time. Which isn’t a bad thing.
But to claim that this amounts to a strategic end-user desktop goal isn’t right. Especially when red hat have come out and said that they feel FDO isn’t worth it as a business.
Doctormo:
“Especially when red hat have come out and said that they feel FDO isn’t worth it as a business.”
Have an authoritative reference to what you are referring to there?
-jef
I thought it was common knowledge that Red Hat didn’t have an FDO straedgy:
http://news.cnet.com/2100-7344_3-5205117.html
Read that, it’s 6 years old and the message hasn’t changed. Fedora is NOT an FDO, it’s a test bed for RHEL.