The Man Also Known As "I'm Batman!"

The Intelligent File Format: Part 1

2006-02-20T21:42:00.000-08:00

Category: Conceptual Design

One of the most frustrating things to any computer professional is the wide variety of file formats. Day in and day out we deal with documents, archives, images, multimedia, and other files that only open in a specific program. We try to make our lives easier by using multi-format programs like PowerArchiver, OpenOffice, GIMP, and VLC. However, these programs often fail to render the file contents accurately. Or worse, we come across an old file in a format that's no longer supported and spend hours trying to find tools to open it.

Perhaps even more frustrating is that many standards have been proposed to solve these issues, but become quickly ineffective as new technologies make the standards obsolete. For example, who is going to encode their video in standard MPEG format when options such as DiVX, Sorenson, and MPEG4 exist?

The problem is even worse for Software Developers who spent inordinate amounts of time reverse engineering formats and code in order to gain compatibility with just one format! Having to reverse engineer a large number of formats becomes an impossible task that encourages developers to find less than ideal shortcuts. (For example, MPlayer redistributes the Windows Media Codecs with a custom linker.)

Some Operating Environments (most notably the Apple Newton) have tried to solve this problem with "standard" interfaces between user programs. This often fails, however, as the programs go out of date, and the interfaces change. What's needed is a way to magically obtain access to the data inside any file. A way to obtain the structure of the data without reverse engineering someone else's parser.

A Bit of History

I've been long impressed by the number of file systems supported by Linux and FreeBSD. Even with incomplete support, these Operating Systems have made life easy for those of us with dual boot systems and multiple drives. Sadly, many operating systems don't reciprocate. Some fail because the developers don't want to support other OSes, but many fail because they can't reciprocate. The filesystem format may be too new, or perhaps the OS is no longer developed for. Either way, getting even a modicum of support seems difficult, if not impossible.

This lead me to think about ways of improving the situation. One Linux project actually used the NTFS driver from Windows to provide write support. Could the concept work the other way around? Could the Linux driver be compiled for Windows or specially linked against? Possibly. Wouldn't it be nice if a standard existed for file system drivers?

Then an idea occurred to me. What would happen if the beginning of file systems embedded a driver for accessing the disk? If the driver was in some sort of neutral format (similar to the X Windows drivers), then any OS could access the file system! In fact, the precise format of a given file system would become irrelevant. You could format your disk to the best file system for your need, and feel confident that it would work in any OS.

Or so the theory goes. I wasn't quite starry-eyed enough to believe that Microsoft, Linus, the FreeBSD Foundation, and Apple would suddenly all become agreeable just because I said so. Still, the concept had merit so I began to work on a prototype for a File-System-in-a-file. My thought was that such a library could prove the concept as well as provide an excellent choice for developers looking to keep all their program data inside a file (ala Access MDB and Outlook PST files) while still allowing the developer to plug in a more robust format at any time. If the concept could be taken one step further, perhaps it could become useful on external Flash drives; many of which are stuck with the sub-optimal FAT file system for compatibility reasons.

The Logical Conclusion

While this concept was exciting in of itself, it didn't even begin to scratch the surface of what was possible. It wasn't long before I considered the fact that a file system is nothing more than a hierarchical database. There's nothing inherently special about it, so why can't the file system payload be replaced with some sort of other data? As long as the embedded driver can read the format and produce some sort of usable data structure, there's no reason why the concept couldn't be extended for all types of data! Images, documents, multimedia, archives, and more could all be converted to self-describing formats.

Of course, like any technological innovation, the concept is not without it's pitfalls. Issues that need to be addressed are:

File Size - Embedding a driver will add overhead to the format.

Upgradability - Files are tied to specific version of the format.

Interface - How do we link the APIs at runtime?

Security - What's to stop embedded code from launching a virus?

Portability - How do we embed code that can work on all platforms?

Performance - How do we provide maximum I/O throughput to files that are performance sensitive?

Let's go over each of these items and investigate the issue in detail.

File Size

There's no denying that a driver in the header would mean an instant increase in file size. For small files, this can easily double the size. It's even conceivable that the embedded driver could be larger than the original file itself!

However, there are mitigating factors to consider:

Disk space is cheap. Adding a few kilobytes per file is unlikely to produce any appreciable increase in storage requirements.

Bandwidth in modern systems is far greater than it used to be. Adding a few kilobytes will not increase transfer times to any noticeable degree.

The driver can be compressed using a standard compression algorithm. This may reduce its size considerably.

Upgradability

Since the driver is embedded with the file, files become tied to the specific version of software that they were written with. If the driver software has bugs, these bugs will continue to propagate as long as the file is in circulation. On the other hand, this also reduces the number of version incompatibilities by ensuring that the original software is always available to parse the file.

Many software packages rewrite files anyway, so this is generally not as big of an issue as it may seem.

Interface

One of the more challenging aspects of this scheme is how to link the driver. One would assume that an Image would have a very different interface from a File Archive. Runtime linking tends to be hard enough without adding completely unknown interfaces to the mix. And how do we document the available APIs to anyone who wishes to load the file?

Thus we need a way to store sufficient meta-data about the APIs to allow for proper runtime linking.

Security

The greatest hazard posed by this format is that it allows arbitrary code to run every time a file is loaded. This potentially makes any file into a potential virus, even if it isn't executable!

What we need is a secure environment to run this code inside of. Such an environment would have to have a foolproof method against accessing the file system, network resources, GUI, and program memory. It can't allow for buffer overflows, and it must be capable of guaranteeing that the file handle passed to it can't be used against the parent program.

Portability

Above all else, the embedded driver must be portable. It does no good to invent a universal file format if it can't leave the confines of the x86 platform. Thus the best solution is to use either a portable scripting language or a language capable of executing on a Virtual Machine.

Performance

With files growing considerably in size, performance has become a major concern. Multimedia files in particular tend to be sensitive to I/O performance, meaning that the scripting language or VM must be capable of using the maximum system throughput without compromising security or portability.

The Obvious Choice

With all these constraints and issues in mind, the choice becomes extremely narrow. Scripting languages like JavaScript and PERL are portable, but tend toward lower performance. Virtual Machines like Smalltalk and .NET have performance, but not high security. The only choice left to us is Java.

The reasons for using Java are:

Security is a core feature, not an add-on. Any chunk of code can be perfectly firewalled off from the rest.

Java is portable to all major platforms, and can be ported to many more.

Java Performance has increased considerably over the years, making it one of the fastest choices on the market. In simple algorithmic usage (e.g. decoders, cryptography, compression, etc.) Java has been shown by many benchmarks to be faster than native code.

Java Reflection makes it easy to load a dynamic library, no matter what its source.

Java can interface with nearly all languages. If you want to use portable file functionality in your C program, for example, there is nothing stopping you from using a JNI interface to load the data.

Java bytecodes are small and compress well. They are regularly much smaller than a comparable native program.

Haven't I heard this before?

As many readers may note. this concept is not without its precedent. Self Extracting Zip files and installers have commonly used a similar technique to distribute their payloads. While the previous concepts have not been quite as far-reaching as what is described here, they are certainly predecessors to the Intelligent File Format.

XML files have also lead the way by encouraging file formats that are common and easy to share. While the idea of a central repository of all XML DTDs and Schemas never came to pass, the overall concept is still running strong and is the basis for many cross platform protocols such as SOAP and XML-RPC.

Go to Part 2 ->

The Intelligent File Format: Part 2

2006-02-20T21:41:00.000-08:00

Category: Conceptual Design

In the second part of this article, I'll attempt to address how the Intelligent File Format might be implemented.

Like most file format documents, we'll begin by describing the physical format of the file, then we'll delve into the contents and how to link them.

The Format Table

As you can see, the format is fairly simple with just a header, the the JAR file containing the code, then the actual data used by the file. In theory, the code embedded in the JAR portion of the code would only ever see the Payload portion of the file.

Header

The header would consist of a structure similar to the one below:

File ID	4 bytes	0xCAFEFEED
JAR Length	4 bytes
Flags	1 byte
Loader Class	UTF8 String

File ID

The file ID is nothing more than a standard identifier used to auto-detect the file type and ensure that utilities know that this is a binary file.

JAR Length

Informs the loader exactly how much of the file is taken up by the embedded software. The loader can then use this information to find the beginning of the payload stream.

Flags

Bit 0 - Write
Bit 1 - Random Access

The write flag tells the loader if the file can be opened for writing, or if the data is considered read only. This is important because if no software to write the file is included, the program using the data may linger under the assumption that changed data has been saved when in reality no actual data has made it to disk.

The random access flag tells the loader software that the file cannot be streamed. If the file is being loaded over the network or other serial location, the program requesting the file will be forced to move the data to random access storage (such as a temp file) before it can be opened.

Loader Class

The loader class is a string that tells the loader software which class to call inside the embedded JAR file.

JAR File and Supporting Classes

The JAR file needs to contain sufficient software to load the file data into a code structure and then optionally write that structure back out.

While streaming of input/output can be handled by the standard java.io.InputStream and java.io.OutputStream classes provided by Java, random access will need a new API. The only API that the Java Virtual Machine provides for random access is the java.io.RandomAccessFile API which is far too file specific to meet the needs of a modern, portable file format. The data could be randomly accessed on disk, in memory, or over the network. Thus a new interface must be developed to replicate the APIs of java.io.RandomAccessFile, but without tying the implementation to any particular storage system.

The following interface and abstract class accomplish that task:


public interface RandomAccess
{
    public long getLength();
    public void setLength(long length);
    public void seek(long position);
    public long getFilePointer();
}


public abstract class RandomAccessData 
  implements InputStream, OutputStream, Datainput, 
  DataOuput, RandomAccess
{
    public abstract boolean isWritable();
}

The "isWritable()" method returns true if the file can be modified. The rest of the methods are equivalent to the methods contained in the java.io.RandomAcessFile class.

The loader class referenced by the header is expected to conform to an interface such as the following:


public interface IntelligentFileFormat
{
    public String[] getInputOutputInterfaces();
    public String[] getOutputInterfaces();
    public String[] getFormatterInterfaces();

    public Object getInput(InputStream in, String type) 
        throws UnsupportedOperationException, FileFormatException;

    public void getOutput(OutputStream out, Object output) 
        throws UnsupportedOperationException, FileFormatException;

    public Object getFormatter(RandomAccess io, String type) 
        throws UnsupportedOperationException, FileFormatException;
}

The first three methods inform the loading software what types objects this code might return. While most implementations would only have a single type of object to return, some implementations may allow for the data to be loaded in a variety of ways. For example, a vector image may be returnable as a DOM of the vector data, or as a java.awt.Image object. Alternatively, a software vendor may choose to fully document a "simple" interface while leaving the more complex, feature rich interface undocumented. (See part 3 for more information on this.)

The type of the object is expected to be a fully qualified class name. The types of objects that can be written are not required to be symmetrical with the types of objects that can be read. Using the vector image example, the vector DOM might be writable to disk while the java.awt.Image object may not.

The latter three methods are where the reading and writing occur. The getInput() method returns an object that may stream the data in, potentially allowing for partial renderings of the data to be shown while the file is still loading.

The getOutput() method uses the object passed in to rewrite the payload data in the file. As stated above, the type of objects that can be written out will not always match the types that can be read in.

The getFormatter() method returns an object that has random access to the payload data. Depending on the setting of the "write" flag and the mode under which the file is opened, the returned object may also be capable of changing the payload data.

If the payload data is not in the expected format or the loader passes an unknown type, the methods will throw a FileFormatException. If the class does not support a method (e.g. getInput() is called on a random access file, getFormatter() is called on a streaming file, or getOutput() is called with an object that cannot be written), a UnknownOperationException will be thrown.

Implementation

An implementation of the above software would carry out the following steps:

Open the file for reading and/or writing.

Read in the header.

Use the JAR size in the header to extract the JAR file.

Load the JAR file into a secure ClassLoader.

Ask the secure ClassLoader for an instance of the class listed in the header.

Get a list of supported interfaces from the class.

Open the payload for streaming or random access based on the flags in the header.

Cast the object returned by the implementation to the expected object type -OR- use reflection to investigate the available APIs.

Go to Part 3 ->

The Intelligent File Format: Part 3

2006-02-20T21:40:00.000-08:00

Category: Conceptual Design

FAQ

Can a producer "cheat" by not documenting their interfaces?

Yes and no. There's nothing in this design to prevent someone from failing to document their interfaces or providing documentation for a crippled interface while keeping the more robust one a secret. However, the matter of reverse engineering becomes orders of magnitude simpler. Rather than trying to guess at the boundaries of fields or decompile a software package, you can now use reflection to investigate and document the classes and methods available in the object representation of the data. In this way, the format can be quickly and easily reverse engineered.

Can the embedded software access a non-payload portion of the file?

If a RandomAccess implementation allowed it to, then yes. Considering the accidental damage this could do to the file, it is important that the RandomAccess implementation passed to the main class be constrained to the payload area of the file. As far as the embedded software is aware the file starts and ends with the payload area.

Won't this technology bloat the file sizes?

It will certainly increase them, yes. However, not all file types are good candidates for this scheme. A good rule of thumb is if the loader software is going to take up more than 20% of the file on average, then the file may be trivial enough to not warrant the use of the Intelligent File Format.

Will this replace all file formats including text files and XML?

No. Most textual formats are already accessible enough as-is. Replacing them with an Intelligent File Format would only server to invalidate the entire tool chain that has been built up over the years. IFF files are much better suited to replacing binary formats such as office documents, where the format is more difficult to reverse engineer.

Can Intelligent Files be embedded inside Intelligent Files?

Many types of documents allow for other types of documents to be embedded inside their data streams. The most famous example of this is Windows OLE (Object Linking and Embedding).

Thankfully, there is absolutely nothing preventing an embedded driver from calling up the loading software and passing in a subsection of its own stream. This would work even with maximum security restrictions in place. However, such implementations need to be careful with RandomAccess files. If the file grows beyond its existing bounds it may overwrite other data in the parent file. Thus more complex implementations would need to consider some sort of paging system for ensuring that growth beyond the end of a file would be relocated to an empty section.

What about temp file access?

As of this writing, I have not fully considered the implications for large binary data files. A common method of handling such files is to swap decoded data in and out of temporary files. However, the security restrictions placed on the embedded software would preclude any sort of temp files from being accessed. Until this is solved, formats would need to page data in and out of the payload area. With CPUs being as fast as they are, and Virtual Memory Paging systems being as sophisticated as they are today, this may not be much of an issue.

Is Java an absolute requirement?

No, it is not. The concept could potentially be done using any Virtual Machine platform. If the use of Java was a problem, a new VM platform could be created to meet the security and robustness needs of this project. However, Java has a decade of commercial development and acceptance behind it that a new VM platform would not. Getting the platform accepted would be a difficult task, especially if it was only useful for loading and saving Intelligent Files.

Does an implementation of this software exist?

Not at the time of this writing. I have built early versions of the portable file system (which do work exceptionally well), but I haven't implemented the Intelligent File Format itself. I did consider doing the work before posting this document, but decided against it for two reasons:

Blogger.com doesn't provide any sort of web space for uploading the software, and I'm not quite ready to make a full OSS or Commercial project out of it.

I wanted to solicit feedback on the concept, and see if anyone has good thoughts on improving upon the design.

Is this design complete?

Most certainly not! While the concept would work fine in theory, there are many tricky aspects to the implementation that need to be worked out. Thus I'd rather start discussion on the concept before taking steps to solidify the concept in code. If you have thoughts, suggestions, or criticisms you'd like to share, feel free to post a comment or email me at akaimbatman@gmail.com. If your comment is interesting enough, I'll even add it to this FAQ.

How do you patch vulnerabilities in embedded JARs? -Cyrus007

The answer to this requires that we first consider the environment that the embedded code will run in. The JAR will be loaded by a secure classloader which will restrict all access to the environment except for a single I/O stream passed in. The permissions to that stream come from a higher level, so the embedded code cannot access other files. Ideally, the embedded code can't access anything other than the payload.

So what are the vectors for attack?

1. The embedded driver could have a vulnerability that could be exploited by data in the file.

Solution: The driver is restricted from accessing anything except the payload data. Thus there's no significant danger to the system. The offending data must have gotten there somehow, meaning that an attacker already had access to the file at an earlier date. Nothing needs to be done.

2. Through an unlikely set of circumstances, data has been downloaded to the file from an external source (such as the Internet) which could exploit a problem in the driver.

Solution: The exploit cannot get anything useful out of the data. It cannot be sent over the network, saved to another location on disk, or otherwise moved out of its sandbox. There's no execution path for non-protected code, either. The only possible application would be to destroy the data inside the file. A rescue utility or patch to the creating program could see to it that these files get updated to prevent data loss. Otherwise, there is no significant risk to the system.

3. There is a flaw in the JVM that allows permissions to be elevated. An attacker could create a virus file that would take over your system.

Solution: There is no logical flaw in the Java security model, so there must be a big oopsy somewhere in the JVM. The JVM needs to be patched against this to prevent both IFF files and Applets from becoming an attack vector. The driver software would remain unchanged.

Beyond those three instances, there's really no other vectors. The code loading the software already has full permissions, so it doesn't need exploits in the driver. Nor can the driver break out of its sandbox without a flaw in the JVM. Since the JVM is the software that must be patched, the driver doesn't need to change. The absolute worst that could happen is that an exploit could destroy the payload for a particular file. Nothing else.

Why not just embed a URL reference to the JAR in question? -Tao

Certainly an appealing design. In theory, the JAR data would only be replicated once rather than with every file. The question this raises is, "What if the JAR is unaccessable?" For example, the website could be down or the user could be on a plane. If he's never opened the file before (assuming a caching mechanism), he'll be unable to access the data. Plus, who is going to guarantee that the driver will remain available far into the future? Will this create a situation where the data must be reverse engineered without any remaining code as reference?

Making the file completely self-contained solves these issues. The driver will always be available, and the data can always be read.

If Java disappears, would the data be harder to extract than in a regular file? -KristofU

No. The embedded driver does not add anything to the payload stream that wouldn't exist without the driver. If the ability to run the code were lost to future generations, the header and embedded driver could be thrown away in favor of analyzing the payload directly. More likely, the executable code would be of help to such "data archaeologists." Even if they are incapable of running the code, it provides a roadmap to the file's contents. As long as they have a basic understanding of the Java bytecodes they can rebuild the driver in a new language. If the understanding of Java bytecodes and contemporary processor design was lost, then nothing is lost to the analysis of the payload. The ZIP markers should help clearly identify the driver's beginning and end points.

-> Digg This Story

China: Getting the Facts

2005-09-29T20:46:00.000-07:00

Category: Commentary

To my faithful readers, I apologize that this week is not about technology. However, I feel that it is important to finally document the abuses of human rights that are applied to the Chinese people every day. There's a lot of misunderstanding on the issues (something I was a part of) and I'd like to set the record straight.

China has been in the news a great deal as of late as they continue to improve their "Great Firewall of China" and work with U.S. companies to restrict free speech. This action has led to a powerful argument about whether China should be allowed to dictate to American companies or not. Should the words "democracy" and "freedom" be removed from the Chinese lexicon?

When I first began paying close attention to the issue, my opinion was "Yes, the Chinese government has a right to control its own borders. They have no law against restriction of speech." That isn't to say that I agreed with the idea of censorship, but it is what the current government dictates, and what the people accepted.

Except for one problem, that is. The people of China never accepted a lack of free speech.

In arguing the point, a very informative fellow pointed me to a copy of the Constitution of the People's Republic of China. Now to be clear, I had never known that the Chinese had a Constitution. I think that it's generally assumed that Communist States simply impose their rule without any true guarantee to the people of their rights. But in 1982 the People's Republic of China (hereafter referred to as "PRC") drafted and ratified a Constitution granting the people very broad rights and privileges.

Free Speech

Probably the most critical right any government can guarantee its people is the right to free speech. It was so important to the United States, that it was made the first amendment to the Constitution. The Founding Fathers knew from early case law and criticism that the only way to keep a nation free was to make sure that the people could know what the government was doing and speak out against it.

Take careful note, however, that I said "guarantee", not "grant". The ability to speak your mind is a basic right that humans have. It can only be taken away, not granted. Besides taking the right away, a government can attempt to protect that right and guarantee that it can always be exercised.

There will always be limits, of course, but it's important to respond to excess "free speech" with an even hand. For example, if I stand up in a restaurant and begin to loudly shout about government injustices, that is out of line. Thankfully, it is likely to only get me thrown out or brought up on a minor charge of disturbing the peace.

Now consider the PRC Constitution for a moment. Article 35 clearly states:

“Citizens of the People's Republic of China enjoy freedom of speech, of the press, of assembly, of association, of procession and of demonstration.”

It also states in Article 41 that the people of China may freely criticize their government in an open forum:

“Citizens of the People's Republic of China have the right to criticize and make suggestions to any state organ or functionary.”

Yet in the news we constantly hear that the words "democracy" and "freedom" are being filtered and removed from the speech of the Chinese people. Blogs that formerly were a good way of telling other countries about the Chinese injustices are now shut down and silenced. As a Chinese citizen you may be subject to arrest and significant jail time if you're caught violating the PRC government's new rules on speech.

Consider the protesters in Tiananmen Square. They were peaceful protesters who wanted an audience with the government. Instead, hundreds (if not thousands) of protesters were killed.

Is this the free speech the Chinese government "guarantees" its people? Is this the right to criticize the government that the Chinese government "guarantees"? It doesn't seem like much of a guarantee at all, does it?

Religion

Another right that the PRC "guarantees" its people in its constitution is the right to practice religion free from fear of government reprisal. Now I understand that many readers may have strong feelings for or against religion. But part of freedom is allowing people to believe and practice whatever they want, regardless of what you think about it. For example, I may not believe in the Muslim religion, nor may I even approve of it. In fact, after the September 11th attacks on my country, it can be very difficult to separate the people who committed the heinous acts from the religion that they purported to believe in.

Yet as a US Citizen it is my duty to protect the beliefs of others, as long as they are exercising personal freedoms and not freedoms over others. (This is an important distinction. If exercising your freedom takes away freedom from others, it cannot be granted.) What that means is that I treat the people who I know are Muslim with the respect they deserve regardless of their religion. And should anyone else attempt to bridge their freedom, I am bound to defend them so that we all can be free.

Now consider Article 36 of the PRC Constitution:

“Citizens of the People's Republic of China enjoy freedom of religious belief. No state organ, public organization or individual may compel citizens to believe in, or not to believe in, any religion; nor may they discriminate against citizens who believe in, or do not believe in, any religion. The state protects normal religious activities. No one may make use of religion to engage in activities that disrupt public order, impair the health of citizens or interfere with the educational system of the state. Religious bodies and religious affairs are not subject to any foreign domination.”

Does it not say that people are free to practice any religion free from government reprisal? Of course, the PRC interpretation is "any religion except Falun Gong". Or Catholic Christianity. Or Tibetan Buddhism. Or any other religion for that matter.

You see, as soon as a follower of a religion does something that the PRC state disagrees with, the entire religion is made illegal. Yes, you heard that right, the entire religion. Personal responsibility for one's own actions is not taken into account. Rather, the religion is immediately blamed and outlawed. In this way, the PRC doesn't have to support any religion they don't want to. Oh, and its for the protection of the people against themselves.

Words Without Meaning

It is easy for the PRC government to justify their position thanks to Article 51 of the PRC Constitution:

“The exercise by citizens of the People's Republic of China of their freedoms and rights may not infringe upon the interests of the state”

In a single paragraph, the PRC declares that the entire document is meaningless, and that rights are granted or revoked at their whim.

But is it really that easy?

You see, the PRC government never guaranteed the people of China any rights, because they never guaranteed that they themselves can be held accountable. And without accountability, their Constitution is just words on paper.

In the US, there are sets of checks and balances that ensure that the governments power can be overturned if abused. The first line of that defense is the court system. If Congress attempts to pass an unconstitutional law, concerned citizens can bring a suit to the Supreme Court to strike down that law. If the President attempts to overstep his authority, the Supreme Court can again rule the actions out of line and command the resources of the US against him.

The second line of defense is the organization of the government and military. Power is distributed throughout the government to ensure that no one person or group of people can wield complete and tyrannical control. In fact the various parts of the government are actually set against each other to prevent consolidation of power. Similarly, the military is intentionally divided, again to prevent power from consolidating and the military taking over.

The third line of defense is the much maligned second amendment. During the revolutionary war, the well armed Minutemen were highly successful in driving back the British forces. The Founding Fathers believed that without easy access to weapons, the US could have never managed to free itself from the British government. So they wrote the second amendment of the Constitution to ensure that the people could overthrow the government should it ever become corrupt beyond repair.

The fourth and final line of defense ends up being the true first line of defense: The freedom-loving people. While everyone likes to disagree on what exactly freedom means and how far it extends, nearly every citizen of the US wants to preserve freedom. On the small scale this means that the people of the US try to balance the system by electing representatives, voting on issues, running for office, and using the courts. On a larger scale, if an all out war started between the military and the people, you can be certain that a large portion of the military would take the side of people.

So be glad for the freedoms you have in the U.S., and in other free countries around the world. And the next time someone tries to tell you that, "there's no law against limiting speech in China," point them to this page and help them learn the facts for themselves.

The author can be contacted at akaimbatman@gmail.com.

Links:

Full Text of the PRC Constitution
China sets new rules on Internet news
Business at the Price of Freedom
Tiananmen Square protests of 1989
Yahoo! Helps Jail Chinese Man
UN Commission on Human Rights Violations in China

Come Dream With Me: Stirling Engines (Part 1)

2005-09-15T21:02:00.000-07:00

Come Dream With Me: Stirling Engines (Part 2)

2005-09-15T21:00:00.000-07:00

Category: Come Dream With Me

The "Come Dream With Me" series is intended to make light of new and promising technologies through works of fiction or direct dictations about where the future might lead. Whenever possible, links are given to support the technology behind the concepts presented in this series.

When we arrived at the garage, Jameson opened the garage door and pointed me to a car inside. It was surrounded by a variety of workbenches loaded with tools. I didn't recognize the make and model, but it was obviously a fairly modern car.

"Here we are," said Jameson. "We got it all put back together for your tour. I'm afraid we like to tinker with it to see if we can improve the mileage."

"This is some sort of custom, high-mileage car?" I asked.

"Sort of. It's actually a modified GM EV-1. The EV-1 was GM's attempt at producing a 100% electric car. To the customers who leased the car, it was a complete success. GM, on the other hand, decided that they would never make any money off the idea and began scrapping the cars. We managed to acquire a few by appealing to GM as a research organization.

"This particular vehicle you see in front of you has been modified from its stock configuration. We've added a Stirling engine to charge the batteries and directly power the vehicle, along with an advanced fuel system that allows it to run on Gasoline, Diesel, Ethanol, Bio-Diesel, Kerosine, and just about anything else we can get to burn with sufficient energy. We can even run the engine on hydrogen or propane, but that would require a very different form of fuel system."

"Wow. So what kind of mileage does this vehicle get?"

"On regular gasoline, we can get 40 miles to the gallon and up. We haven't gotten a proper certification on it yet because we're still working, but it's definitely more fuel efficient than a regular car," he answered.

"That doesn't sound too much better than today's hybrids," I commented.

"True," Jameson responded. "We're working to improve the engine efficiency, and hope to significantly increase the mileage. However, the real key to this vehicle is that it can accept so many forms of fuel. Petroleum products are becoming far too expensive to maintain the economy. The problem is that hydrogen fuel cells are still a long way off, and existing vehicles need special conversion kits to use a fuel other than gasoline. Since it's unlikely that we can expect all the gas stations to start supplying new fuels tomorrow, we need a plan that can phase out gasoline and phase in the alternatives.

"At the moment, ethanol looks like our best bet. It's a little less energy dense than gasoline, but not significantly so. And its per-gallon cost is competitive with gasoline's current prices. If we can encourage car owners to move to ethanol while still supporting gasoline for when it's unavailable, then we may be able to significantly reduce the price of gas as well as the price of ethanol. And for the long term, ethanol is fully renewable. All we need to do is farm more land."

"But doesn't ethanol cost more energy to farm than it produces?" I inquired.

"No, that's old data. Ethanol became energy positive sometime in the 70's when diesel tractors and better farming methods replaced the old gasoline driven tractors. There are a few researchers who cling to the old figures, but the consensus is that ethanol produces a significant energy surplus." he replied.

"Interesting. So what is the performance on a car like this? You said before that this car doesn't have the same response problems as the engine in the plane. How does it compare?"

"Tell you what," he offered, "why don't you climb into the passenger seat and I'll show you?"

"Alright," I agreed.

Inside, the car looked much like any late model sedan. It had all the standard features one would expect, such as a CD Player, radio, air conditioning, heater, front bucket seats, etc. Other than a small computer readout above the radio, I couldn't tell that it was anything other than a normal car.

"What's that?" I asked, pointing to the digital readout.

"That's a debugging tool we use for development. It shows the charge of the whole battery pack, monitors individual battery cells, reports the power output of the engine, informs me of power distribution, and keeps logs of the mileage attained. I know, it looks like something out of Knight Rider," he said with a smile.

Jameson had no key in his hand to start the engine. Instead he merely depressed the "start" button located near the stearing wheel. I didn't hear anything, but several status lights came on and began reporting information. Jameson eased the car out of the garage and onto a test track that surrounded the runway we took off from.

"Ready?" he asked.

Before I could ask for what, Jameson slammed his foot on the pedal, and we took off like a shot! I looked over at the speedometer and watched it climb smoothly toward 60 with no apparent pauses from shifting. Within barely a few seconds, Jameson leveled the car out at 65 miles an hour as we raced around the test track.

"Wow," I mouthed as I let a breath out. "This thing can certainly take off!"

"Yes it can," he replied proudly. "All the power is shunted through electric motors. Unlike internal combustion engines which are tuned to key areas of torque and horsepower, electric motors can produce nearly the same amount of torque across their entire range of operation. Which means not only a fast zero to sixty acceleration, but also a fast sixty to one hundred acceleration. Now the two don't quite compare as greater wind resistance requires more horsepower to overcome at higher speeds, but electric motors certainly have a lot more kick than direct motive power transmission.

"The same technology is used in most Diesel engine trains. The diesel engine is capable of producing power output well above 400 kilowatts (ed note: that's 536 horsepower we're talking!), but gearboxes are unable to transmit that much power to the wheels without self-destructing. The solution was to convert the rotational energy into electricity, then feed the electricity directly to the wheels. This allows modern diesel engines to smoothly apply power to prevent wheel slippage, like you see in the old movies with steam locomotives, and allow fast acceleration. I don't know if you remember from the last train you took, but I think you'll find that they are quite zippy for massing hundreds of metric tons when pulling cars."

"So the torque is more important than the horsepower?" I asked.

"Sort of," he replied. "Torque is a measure of how much horsepower is applied to the rotational axis. In cars, the ability to accelerate quickly tends to be more important than the maximum horsepower that the engine can put out. For most drivers, 50 horsepower is plenty. The problem is that we put 250 to 300 horsepower engines in cars to make up for the slow torque build-up curve of internal combustion engines. A more powerful engine can apply more torque on demand. Still, that's pretty hard on an engine and seriously reduces its life-span. That's why most engines only run for 100,000 to 200,000 miles of the car's lifetime.

"Now for towing, the maximum horsepower can often be more important than how fast the torque can be ramped up. Or in other words, I purchase a pickup truck to haul a 20 ton vehicle, not win drag races. In those cases, the slower acceleration produced by the slower ramp up of torque is less bothersome.

"So in the end, the horsepower is important. What's more important, though, is whether it can be used when it's needed. For road cars the answer has been 'no'. Yet the automotive engineers have done a bang up job of making the public think the answer was 'yes'!"

"You mentioned that the average driver only needs about 50 kilowatts of power. How much power does the Stirling engine in this car put out?" I asked inquisitively.

"Well, up until a few moments ago it wasn't putting out any power! We were driving entirely from battery power. As the car sensed the battery levels dropping, it ignited the Stirling engine and brought it up to speed. Right now," he said as he glanced at the debugging display, "it's producing its optimal 55 kilowatts of power. About 22 kilowatts are being directed to maintain our speed, while the remaining power is being used to charge the batteries. I'm afraid that I put something of a strain on the battery packs with that acceleration stunt. It topped out at about 90 kilowatts of power. Had the Stirling engine been running, it could have provided further power. Unfortunately, the motors can't accept much more than about 100 kilowatts of power, so the acceleration wouldn't have been too much more impressive."

"That's really fascinating," I commented. "What does GM think of this concept?"

"GM is interested, but they're keeping to themselves on this one. They did a few hybrid conversions themselves and decided that it wasn't worth the effort," he replied.

"Could you expand on that?"

"Sure. In the early to mid 90's, GM attempted to build a Stirling hybrid car. Apparently the program met the cleaner exhaust goals they had laid out, but apparently it failed to reach the thermal efficiencies they were looking for. I'm still unclear on what efficiencies they were looking for, so it may be that their failure was simply that the Stirling engines they tested weren't significantly more efficient than the gasoline engines they produced.

"In my case, I'm unconcerned about far greater efficiencies. The primary goal I'm trying to reach is a car that can phase through the alternative fuels coming to the marketplace. As long as the car performs on par or slightly better than existing vehicles, I'm happy.

"To get back on topic, though, GM tried for another type of hybrid in the late 90's. They took one of the EV-1 cars they had, the same type that we're sitting in right now, and added a gas turbine to provide hybrid power. That's where I got the idea of using an EV-1 for my research.

"As far as I know, the program was a complete success. Not only did it produce significantly lower emissions, but it got 60 miles to the gallon. As a result, GM was able to fit the car with a mere 6.5 gallon tank, giving it a range of 350 miles.

"GM never produced a commercial model of the gas turbine hybrid despite its seeming advantages. I don't know for certain why GM abandoned the technology, but my guess is that the turbines were simply too expensive. Chrysler messed with the idea for 30 years, convinced they could find a materials technology that would bring down the price. As you can guess, they never found a solution and abandoned the tech in the 80's. I have my doubts about anyone ever reviving the idea except for special cases such as military HumVees. Perhaps even some busses could afford to fit the engines."

"Interesting," I said. "And you believe that you won't have the same materials problem with Stirling engines?"

"No, not really," he replied as he steered the car off the track and toward where I was parked. "Stirling engines don't really push the limits of materials in any significant way. In fact, it's quite possible that they would have replaced steam locomotives had useful steel been invented earlier. Sadly, the iron technologies of the late nineteenth century were unsuitable for a proper Striling engine, so the idea was shelved. The real pain with Stirling engines is designing a properly tuned engine. Internal Combustion Engines have seen billions of dollars poured into their development. As such, they've become incredibly efficient and responsive for such an otherwise poor technology. Stirling engines have seen very little development in comparison, so we're trying to jump the gap to produce an engine comparable with what the auto manufacturers produce today.

"Well, here's your stop. I hope you enjoyed your visit."

"I enjoyed it very much, thank you. You've given me quite a bit to write about."

As I got out of the EV-1, I couldn't help but think that Jameson's work may forever change the world as we know it. If his technology succeeds, the world will finally be free from the bondage of ever-disappearing oil and high gas prices. Even better, the adoption of the Stirling engine could result in research and development that could produce more efficient engines than ever before. A new, untapped road for automotive vehicles. I hope he does succeed.

The author can be reached for questions, comments, and suggestions at akaimbatman@gmail.com.

Links:

Wikipedia: Stirling Engine
Why Aviation Needs the Stirling Engine (Homepage)
STM Stirling Engine used in the GM HEV
Report on GM Stirling Vehicle
Ethanol Energy Balance (Homepage)
More Information on the Ethanol Energy Balance Debate
GM Turbine Car
Wikipedia: General Motors EV1

The Synergistic PC

2005-09-01T20:50:00.000-07:00

Category: Concepts

If there's one thing I've learned about Bill Gates, it's that he's a horrible predictor of the future. He may not have actually said that "640K is enough for everyone," but he did make a few other whopper predictions. Among them was his "vision" for the future of the Personal Computer as laid out in his book, The Road Ahead.

According to Mr. Gates, your computer was destined to merge with your television, and the TV would become a super-entertainment system of the future. Of course, this prediction conveniently ignored the technological problems and consequences, not the least of which was that the TV was a blurry, low resolution device. Even if he assumed that the future would produce high resolution televisions, there were simply too many problems with the idea of the television as the home computer. (As Gateway can attest from their failed experiments with the idea back in the 90's.)

Yet there is a certain allure to the idea of merging the television and the computer into one device. Especially when you consider the number of computers now equipped with DVD drives, and the sheer amount of illegal file sharing of movies and television programs. Perhaps the TV isn't going to become your computer, but there certainly is a demand for the internet to merge with your television.

Gates Got it Backwards

Five years ago I argued with my wife about the getting a television. She was against the concept because she was worried that the television would always be on, thus distracting the family from spending time together. Yet I had managed to pique her interest in Star Trek, and she wanted to be able to record and watch the old Star Trek episodes as much as I did. (Believe it or not, there wasn't really anything else on that we wanted to watch.)

So I managed to cut a deal with my wife. We'd get a television card for the computer, and use that in conjunction with super-basic cable (~$14.95 in California) to record and watch our shows. She agreed, so I purchased and installed the TV card. With the help of the free version of SlipStream PVR, we were soon recording all the Star Trek we wanted.

Which brings me to my point. Mr. Gates got it backwards. Televisions weren't going to become the computer, the computer was going to become the television! Even in '96 (when Mr. Gates published his book) this was apparent. TV Cards were affordable, and computers were capable of higher quality pictures than the "normal" TV screens of the day. Why would anyone bother merging the computer into a massive vacuum tube when they could just wait for better computer monitors?

So What is the Future?

Consider the following configuration for a moment:

Modern computer with a 250 GBs or more of SATA-300 Disk
30 inch LCD TV with DVI inputs
TV Card with Remote Control
5.1 Sound Card and Speakers (probably built into the MotherBoard)

Save for the 30 inch LCD TV, that should sound like a fairly normal PC today. The TV Card with the Remote might not be "standard", but it would add less than $50 to the price of the PC. But what about that 30 inch TV? What is that for? If you're thinking, "TV/OUT", then you're thinking wrong. An LCD TV with a DVI connector can actually act as the computer monitor, because it is a computer monitor. Just mount it on your wall, and be free from eye strain forever!

As cool as that is, though, it's still not my final point. The final point I'm getting to is in connecting the power of these few components. Consider for a moment, this little configuration could replace your television, your TIVO, your CD Player, your DVD Player, and even your stereo system! (Especially if you opt for the FM tuner in the TV Card.) All of these devices combined into one unit! You can record your favorite shows, pause live TV, watch your rented movies, listen to your tunes, everything!

Of course, if that was it then such a machine would be simply "nice to have". Certainly not a requirement for any household. Especially houses that have already invested in a TIVO, Stereo, DVD Player, and CD Player.

Consider this for a moment:

With this setup, you can not only do the things I've already listed, but you can also watch movies and listen to music directly from the internet. Through iTunes, you can instantly download new music and listen to a huge selection of radio stations. Through your web browser, you can pull the latest PodCasts and listen to news from around the world. Through MovieLink you can pull the latest movie releases. And if this catches on, you might even be able to watch your favorite television shows! Plus you can still use your computer for all the office and gaming work it performed before!

In other words, computers have finally reached the state of Digital Convergence. All the functions that we used to have to build separate devices for are now nothing more than simple inputs or outputs from our computers. Thus we are able to strip electronic devices down to their simplest level and combine all the functions together to produce something that is more than the sum of its parts.

When Can I Get This Cool Stuff?

Now, of course! The stage of convergence I just talked about is here today! All the technology can be had for a reasonable price!

To help you build your very own convergence machine, I've compiled a list of parts and software that could help you build it from scratch. These parts attempt to combine good functionality with a good price. You can use the parts I suggest, or find equivalents that suit your preference.

Hardware Links

Leadtek TV Card - Contains a TV Tuner, FM Tuner, remote control, plus fully featured PVR software with live TV pausing.

A8N-E - A motherboard with AMD64 support, onboard 8 Channel Audio, PCI-Express support, 10 USB ports, plus support for up to 4 SATA-300 drives. (I recommend that you also get this board with a CPU and Memory. AMD64 3500 and 1GB is my choice.)

NVidia GeForce 6600 - All the Modern 3D Features, High Performance, DVI Out, Supports Linux, Dual Head Support, Good Price.

Western Digital 160 GB SATA Drives - Good Price, High Performance, Reliable Company. (I recommend at least 2 drives for PVR work. The motherboard I recommended supports up to four, so go hog wild!)

Case - Well, you need something to put this equipment in, don't you?

DVD Drive - Burns DVDs (+/-/RW), CDs, and can even label disks!

Speakers - All in favor of sound, say 'Aye' in Dolby 5.1 Surround.

LCD TV - Probably the most expensive part of the purchase (up until now, you shouldn't have racked up more than about a thousand dollars), but it's the centerpiece that makes the entertainment center complete. If you don't actually need a big TV, a 19" LCD Monitor will always work too.

Software Links

MovieLink - Rent Movies Online. Seriously. (The down side is that most new releases are only available for a short time.)

iTunes - Music, Radio Stations, etc. You know the drill.

Windows XP or Linux/BSD - Windows has better support for the bundled PVR and Remote Control software out of the box. Linux and BSD require a bit of tweaking before they will operate at full capacity, but there are plenty of options.

Looking Forward

Believe it or not, there is still more convergence on the way. For example, shouldn't it be possible to combine the game console with the computer? How about the problem of computing while watching the television? Well, technology isn't there yet. Keep an eye out for a future series where I present a vision that would further meld our homes into our computers, thus reducing components and increasing features.

Tell us about your multimedia powerhouse! Post a comment for all to read, or send a letter to the Bat at akaimbatman@gmail.com

What's Old is NeWS Again

2005-08-18T20:24:00.000-07:00

Category: Commentary

Nearly twenty years ago now, the father of the Java programming language spear-headed a rather different idea for its time. GUIs were still in their infancy, but James Gosling and David Rosenthal envisioned a system whereby rich applications could be seamlessly delivered over the low-bandwidth networks of the time. Even better, the system was designed such that if you could see it on the screen, you could print it. That system was called New extensible Windowing System, or NeWS for short.

Exploring NeWS

Many people don't realize it, but PostScript is a fully featured, Turing complete programming language in addition to being an excellent method of drawing images on pages. This makes it uniquely suited for both screen and print work. NeWS relied heavily on the PostScript language by delivering system-independent, PostScript applications from the server to the client.

NeWS contained special PostScript libraries that allowed a downloaded application to communicate with the server. Via this linkage, the PostScript program could access data and resources over the network. This form of network communication would be later seen in Java Applets.

How Did NeWS Differ from X-Windows?

The primary difference between the X-Windows model and the NeWS model was that the NeWS model pushed the entire interface to the client, whereas the X-Windows model only pushed drawing commands across the network. The end result is that the NeWS model tended to be far more network friendly. Images would only be pushed once in NeWS, and the minor amounts textual data transferred meant that bandwidth was conserved. Under X-Windows, image data may be transferred many times. Especially if that data is embedded in the interface.

In addition, NeWS did not require that user state data such as the clipboard contents and mouse movements be transmitted over the network. This improved security since only the information that was absolutely required was transmitted. It also reduced the potential for attackers to secretly connect to a "graphical server" to obtain such information.

What Happened to News?

For all intents and purposes, NeWS was superior to X-Windows. So why did X-Windows become the Unix standard while NeWS vanished into the annals of history? The answer lies in the advanced technology used in NeWS, and the ideas the software industry of the day had about profitable ventures.

You see, when Sun created NeWS they charged a licensing fee to developers for the use of the technology. This wouldn't have been so bad if Adobe didn't also charge a licensing fee for the PostScript technology. Not only that, but PostScript turned out to be a very difficult language for developers to work with due to its Reverse Polish Notation. Combine this with the fact that NeWS was not an open standard, and you end up with a lot of customers who wanted something a bit more open and less expensive.

As a result, the rest of the Unix industry jumped on the freely available X11 standard in an attempt to prevent Sun from gaining a further technological lead. The ploy worked, and very little software was developed for Sun's NeWS environment. Over the next few years, the Unix industry would develop the CDE desktop which only required licensing fees for programs that used the Motif library.

AJAX is NeWS Reinvented

If the concept behind NeWS sounds vaguely familiar, it should. The development of the HTML, CSS, and JavaScript technologies has resulted in a system very similar to that of NeWS. Very rich applications can be delivered securely over the Internet thanks to these technologies. The introduction of the XMLHttpRequest component only serves to improve upon this model. If the browser base can be made standards compliant, developers will finally be able to realize the dream that NeWS started.

Err, What is AJAX?

AJAX is a market-speak name given to the combination of HTML/CSS/JavaScript technologies. It usually implies the use of the XMLHttpRequest component, but not always. The idea behind the AJAX concept is to deliver rich applications by asynchronously requesting data from the server. That data is then inserted into the user's view by directly modifying the HTML document. The modifications are usually made via the JavaScript DOM API. CSS is then used for layout and special effects.

For example, an AJAX application may present a set of clickable "icons" to the user. When the user hovers over one of these icons, it is highlighted to let the user know that it's clickable. The highlight effect is usually achieved either through the CSS :hover attribute, or via the CSS "class" being changed on mouse over.

When the user clicks on one of the icons, a new DIV pops on top of the existing text. A request is then made to the server for the information to populate that DIV. The information is inlined into the DIV via the JavaScript DOM API. The popup DIV might allow scrolling by setting the "overflow: scroll" CSS property. Most likely, this DIV would contain a close button that would remove the DIV and allow the user to select another icon.

Is AJAX the Future of Applications?

Yes, and no. While AJAX places a great deal of power in the hands of developers, it does fall short in many areas. For example, an early demo for NeWS was an online Pizza ordering application. The interface had a "preview" of your pizza on screen, very similar to the shaped windows of today. The problem is that HTML/CSS currently provides no method for creating shaped objects on the screen. A common workaround is to "fake" shapes through the use of images with transparent regions. However, this is not a general purpose solution and often creates many problems when trying to create common UI components such as boxes with rounded corners.

The upcoming CSS standards attempt to address these issues by adding more styling attributes. Yet that still leaves us with more issues. Scaling is only supported on certain components, and rotation is not supported at all. These features are currently seen as unnecessary, so there are no plans to add them in future CSS revisions. In fact, the W3C would probably direct you to a completely different standard: SVG.

Scalable Vector Graphics (SVG for short) is a vector drawing language similar (but not equivalent) to Macromedia Flash. It allows for arbitrary images to be drawn on the client, providing a method through which more advanced interface components may be embedded. Scaling and rotation are not only offered, but we could draw, spin, and shear our pizza! By mating SVG to the HTML/CSS/JavaScript stack, we now have a complete technology solution for producing nearly any application desired.

The raises the question, however: Is it really feasible to generate SVGs on demand? The answer to this is a bit unclear at this point, but there is precedent. In 2002, programmer Lee Semel took advantage of another text format that's supported by most browsers: XBM. XBM is a black and white format originally designed for early X-Windows programs. Images are represented by textual plottings of the image data. Since this data was intended for embedding directly into C programs, the format appears to be C code.

Many web browsers allow for the source of these images to be accessed and even changed. Mr. Semel used this to his advantage when he created his JavaScript program. What program was it? What would any red blooded coder write if he was given a raw frame-buffer? A First Person Shooter, of course!

I love hearing your opinion! Send a bat-a-gram to akaimbatman@gmail.com!

Links:

Wikipeida Entry for NeWS
NeWS Information Site
Wikipedia Entry for AJAX
Ajaxian
James Gosling

Free Books on the Internet

2005-07-29T20:10:00.000-07:00

Category: Cool Stuff

Over the years, one of the things I've noticed is a constant decline in the quality of books. Where shelves were once filled with books on OS Design, Compiler Theory, 3D Graphics Algorithms, and Basic Data Structures, now they are filled with books on Learning Java in 24 Hours and Idiots Guide to OS of your choice. While this decline is primarily due to the flood of poor programmers who entered the market during the boom, it has not abated during the bust. More books on 100 Things You Should Already Know about API XYZ continue to hit the shelves while the classic material gets more and more buried.

Thankfully, the Internet has taken the place of these once cherished hunks of dead tree. You may not be able to find a good book on Text Parsing in Barnes & Noble, but you can find all the same info plus someone to answer your questions on the internet. Still, sometimes it's nice to have a well written book to read.

Well have no fear! Many of those books we once cherished are now available on the Internet, and I've compiled a list for you of some of the more interesting ones. Nearly everything below is available in PDF format. I actually skipped over some of the HTML only books, because I very much like the idea of having a file that can travel with me. HTML is far too unwieldly. So download a book, curl up in front of the fire, and get a tan in the rays of your LCD screen!

Advanced Linux Programming - Pretty much everyone knows the standard C libraries, and the POSIX libraries are a requirement for any serious Unix developer. If you want to read a book on beyond such simple programming APIs, then this is the book for you. Processes, Threads, IPC, I/O, Assembly, it's all covered!

Bruce Perens' Programming Series - Bruce Perens is a very interesting (and probably quite busy) guy. So busy, in fact, that he's single-handedly written dozens of books on advanced programming topics! Ok, so maybe he didn't write them all. Don't let that deter you, though. This series of books cover everything from Java to Mozilla Platform Development, and everything in between!

Parsing Techniques - As a more junior developer, I noticed that nothing quite honed a developer's programming skills like trying to build a better text parser. This book attempts to break down many of the techniques that have developed over the years, and establish the theoretical foundations behind them. A must read for any serious developer!

Creating XPCOM Components - With WebApps developing into a more and more powerful solution for deploying applications, it's no wonder that interest in the Mozilla Platform is on the rise. Most people think of Mozilla as just a web browser, and take no note of the amazingly diverse set of APIs that are used under the hood. This book provides much more depth and info on the Mozilla Platform, and explains the XPCOM structure in depth.

Practical File System Design with the Be File System - I'm only going to say this once: Download this book NOW. This book is the definitive guide to file system design, and was written by the guy who designed the BeFS Database File System. If you wonder why his musings on a dead Filesystem for a dead OS are of interest, just consider this: He works for Apple. Can you guess which components he developed?

Thinking in Postscript - As programmers, we like to think in pixels. Pixels are a discrete, indivisible unit that are easy to calculate and plan for. Yet we work with them day in and day out without ever realizing the limitations of such units, Nearly every programmer has one of those "Ah ha!" moments when he's first introduced to vector graphics. Make your "Ah ha!" moment happen today. Download this book!

Introduction to Machine Learning - Artificial Intelligence. Talk about an area that has had tons of money poured into it with very little practical return. Even the Spam filters in common usage today tend to use statistical models instead of AI! Yet the field is very interesting, and the constant march toward faster computers makes home research possible. So pull up a chair, and prepare to dissect your own brain!

A Practical Theory of Programming - Impress all your friends and neighbors! This book is crammed full of mathematical formulas that describe how programming works. It may not teach you how to write LISP, but it will at least give you the math to understand it! (As if anyone understands LISP. I kid, I kid!)

Information Theory, Inference, and Learning Algorithms - Information Theory has become a hot topic in the physics world of late. You see, it turns out that many of the mysteries of the Universe are tied to the ability or inability of particles and energy to carry information. For example, it is perfectly possible to violate the speed of light. Quantum Tunneling is a perfect example of this. Yet it is still impossible to transfer information faster than the speed of light! Is it a universal conspiracy? Who knows? What I do know is that having a good grounding in Information Theory is good for Computer Scientists, Physicists, and your average Developer alike.

Compilers and Compiler Generators - Ever written a compiler? No? Well, one day you may just find yourself in the position of doing exactly that. And when that time comes, you want to be ready. So read this book and understand the theories behind translating source code to machine code.

Linux Device Drivers, 2nd Edition - So you've got OS Design down pat. Plus you've got compiler design down. You've even taken AI and Information Theory! But what about those pesky hardware devices? How in the world do they get controlled? Well, pick up this book and find out!

Linux From Scratch - While abstract knowledge of OS Design is all well and good, sometimes you just need to dive into a working codebase to get a feel for things. While Minix is a great place to start, this book (and LiveCD!) will help you setup a custom Linux machine in no time flat! Move over Gentoo, I'm compiling my own system from scratch!

Now for Some Entertainment

If you managed to plow your way through all of the technical books above, then you deserve a break! Here's a few books and links to entertain your brain!

The UNIX-HATERS Handbook - Don't you just love how fast Unix boots? After all, things crash a-plenty, so it's a good thing it boots fast! Left over from a time now gone is the UNIX-HATERS Handbook. From the days when LISP machines stomped the Earth, and Macs were changing the face of computing (Wait, isn't that happening again?) comes a humorous look at all those little things that constantly go wrong in the OS we call Unix. You'll laugh because it's funny, but you may also learn a thing or two about what the world of computing once looked like.

Baen Free Library - Science Fiction and Fantasy. All free for the taking! Think I'm kidding? Click on the link and find out! My personal recommendations are: On Basilisk Station, 1632, and Mutineer's Moon. Odyssey is a great read as well. Enjoy!

Star Dragon - Star Dragon is a hard Sci-Fi novel from author Mike Brotherton. I can't tell you if it's any good, but I can tell you that it's free. Never hurts to scope out new authors!

Baen Book CDs - So you thought the Baen Free Library was cool, did you? Well so did Mr. Baen. He thought it was so cool that he'd try attaching a few CDROMs filled with even more books to his hardcover novels. The only catch? You can give the CDs to all your friends and neighbors. Wait, that's not a catch, that's a feature! Enjoy these hundreds of books offered at no charge, and with no DRM!

Audio Books for Free - Sometimes the pressures of life get in the way of reading. For those times, there are audio books. But who has a tape player anymore? This site is chock full of free MP3s that you can download to your player and listen to on the go! Sure, it's mostly classics. You do like classics, right?

Project Gutenberg - Speaking of classics, how can any discussion of free books fail to mention Project Gutenberg? With over 16,000 books transferred to electronic format, there's no better place to catch up on your H.G. Wells or Jules Verne!

Linux Needs More Distros!

2005-07-21T20:28:00.000-07:00

Category: Commentary

In a recent article, author Sal Cangeloso suggested that Linux needs to consolidate into fewer (perhaps even one!) distributions. His reasoning behind this recommendation is that the Open Source Community is unfocused in its goals and is wasting time on multiple distributions. He then cites the Mandrake and Connectiva merger as something he'd like to see more of.

While I agree with Mr. Cangeloso on the Madriva merger being good for the Linux community, I disagree with his logic on why it is good.

Multiple Distros Are a Strength

In the mid to late 90's, the GCC project began to stagnate as tight controls were placed on code contributions. Many GCC developers believed that it needed to be reorganized around a more flexible design, and tried to get support from the FSF. The FSF unfortunately ignored these pleas, so a handful of developers used their GPL-granted rights to fork the project into a technology called EGCS (Experimental GNU Compiler System).

Thanks to optimized code generation, better cross-platform support, and the unification of computer languages compilers, it wasn't long before EGCS began to displace the official GCC branch as the developer community's compiler of choice. The GCC developers eventually realized the power of EGCS and merged EGCS back into the main branch.

The point I'm getting at here is that a successful project is going to tend to become risk adverse. It's not that they don't want to move forward, it's more a matter that they don't want to fail and fall behind. This is a perfectly natural reaction. The loophole that exists to escape these sorts of politics is the ability to fork.

By forking, a developer can explore new paths without placing the primary project at risk. If the fork is successful, then it can be merged back into a more comprehensive project. If it fails, then far less was bet on its success.

As a result, the plethora of distributions is actually a strength of Linux (and the BSD community!), not a failing. If everything follows its natural course, distros will come and go, but a few main trunks will continue to benefit from the development. But why do we need multiple main trunks?

Multiple Product Lines are a Strength

If you've ever bought a car, you have probably realized that auto manufacturers sell cars under several different brands. GM, for example, sells inexpensive Chevy cars, high-performance Pontiacs, luxury Cadillacs, powerful GMC trucks, and experimental Oldsmobiles. While these lines each produce very different vehicles targeted at different markets, they are able to share chassises, parts, and research. This lowers the price while raising the quality of the vehicle.

A similar concept can be applied in software. For example, Apple produces a consumer version of Mac OS X and a server version of Mac OS X. The consumer version is tuned to multimedia, laptops, power management, and other features that end users expect, while the server is tuned to heavy networking, throughput, remote installation, and other features that admins have come to expect. The same is true of the Windows Operating System. (Although the differences between some versions were artificially added by Microsoft.)

In the Open Source world, even more flexibility is offered. Here's a quick rundown of distros, and where they have chosen to position themselves in the market. (No, this is not a comprehensive list. So don't complain about your favorite distro missing.)

RedHat: Long considered the leader in Linux development, RedHat has chosen to target the x86 server market with its product. Its spinoff (Fedora) is where the bleeding edge in Linux technology is tested before being integrated into RedHat's product line.

SuSE: SuSE has used its extensive experience in producing a high quality desktop distribution to position itself as the de-facto Corporate Desktop/Workstation distro.

Ubuntu, Mandriva, Xandros, Linspire: These distros are competing in the massive consumer desktop market. The differences between them can be extensive, the least of which is their choice in Desktop technology. Some choose to present users with GNOME, while other choose to present KDE. Each has its own method of easing the user's difficulties with installing software and interoperating with Windows machines and programs.

Gentoo: Gentoo has positioned itself as a powerful build system for constructing custom operating environments. While it's definitely not a choice for your average user, power-users such as technology workers, engineers, and scientists can customize the system to meet the precise needs of their work.

Yoper: Yoper has positioned itself as the "sports car" of Linux distributions. It's compiled to be faster than most distros, and is carefully tuned to meet the needs of most enthusiasts.

Knoppix: Designed to be the ultimate in demoing and portable desktop technology, Knoppix has evolved to allow users to take a familiar environment with them no matter where they go. Combined with a removable media such as a USB Pen Drive, Knoppix can give users the freedom to roam with their files from computer to computer. All without changing a single configuration option on those computers!

As you can see, the Linux community isn't just "wasting its time".

Aren't You Being Hypocritical?

One of the questions that I realize will come up in response to this article is, "Aren't you being hypocritical? You just told us that we need a standard Linux Desktop across all distributions!"

Let me put your mind at ease. I stated no such thing, nor am I planning to state such a thing. The previous article was called The Linux Desktop Distribution of the Future for a reason. A new distribution to test the ideas I presented could only be good for the Linux community as a whole. After all, why force Linux into a box? Why must packages be the perfect solution? Why must we deal with traditional file systems? Why must we stop moving forward?

I realize that many readers are used to wholesale Linux bashing from writers such as myself, and have instinctively reacted to what they perceived to be more Linux bashing and "make it like Windows" solutions. Put your mind at ease. I don't want to make Linux like Windows. I want to take Unix systems into the future, hopefully before the competition gets there.

So, let me ask you this question: What do you want out of a Linux distro? Or more precisely, how would you like to fork today?

You can email the Batcave at the address akaimbatman@gmail.com.

Links:
EGCS History
General Motors Automobiles

RedHat
SuSE
Ubuntu
Mandriva
Xandros
Linspire
Gentoo
Yoper
Knoppix

Linux Desktop Distribution of the Future Follow-up Part 2

2005-07-14T20:12:00.000-07:00

Category: Commentary

Part 1: Clearing Up Misconceptions
Part 2: Refining the Ideas

Some argue that it's easy to make suggestions when you aren't the one figuring out how these suggestions will work. The problem with this argument is that without communicating these raw ideas, the implementation can never be found. "Communication completes thought" is a wise saying that a former boss of mine drilled into my head. By communicating our ideas we often think through the implications and decide exactly how they might be implemented.

As such, this week's article will attempt to present some of the refinements and more detailed explanations that were produced through conversations over the original series. We'll start by going over the tools of the trade.

FUSE

FUSE (Filesystem in USErspace) is probably one of the coolest little inventions ever. By implementing a simple interface, a developer can quickly and easily build a custom file system that runs entirely in user-land. Effectively, this makes Linux act like a Micro-Kernel with only the important I/O running at the Kernel/Ring 0 level.

The advantage to using FUSE (other than to speed the development of a new file system) is that all the libraries that are available to user programs are available to the file system. If you've never worked with Linux Kernel Modules before, allow me to assure you that this is a huge step. Software that runs inside the kernel can only access APIs running inside the kernel. This can limit a developer and make programming quite tricky.

The downside to using FUSE is the same downside that exists in nearly all micro-kernel designs: It's going to be a smidge slower than a kernel level driver. How much slower depends upon the application, but not so much that it would generally be noticeable with modern hardware.

FUSE's advantages have led to its use in SSHFS, GMailFS, and several other interesting file systems.

Berkeley DB (BDB)

Berkeley DB is a very powerful embedded database library. Instead of taking the design to the level of an SQL server, BDB provides only data storage and file indexing. Concepts like a high level query engine and robust network support are left to the developer. This makes it a perfect base for a custom database. An example of this is the Berkeley DB table type in MySQL, which is built on the BDB database software.

AppDisks

One of the key concerns about the AppDisk scheme I proposed was the issue of limited loopback devices. A standard kernel is limited to a mere 8 loopback devices, which could cause a few problems if the user wants to run more than a few applications. While there actually exists a patch to allow for more than 256 loopback devices (!), it's important to understand why a loopback device is needed in the first place.

You see, there's nothing in the mount system that prevents an existing file from being mounted into the file system. As far as it is concerned, a file in /dev is just the same as a file located in your home directory. The reason for the loopback device has to do with the file system drivers. Most file systems specify that they will only mount a block device. As a result, the kernel rejects attempts to directly mount a file as a new file system. What the loopback device does is that it creates a "fake" device that makes a file look like it's a block device.

The loopback device can be eliminated entirely if a file system is used that doesn't require a block device. For example, a "simple" FS could be created for most applications that don't require write support. This FS could be as simple as a TAR or ZIP file, although access times would be quite slow with both of those options. A file system driver could then be written for that "simple" FS, allowing for a virtually unlimited number of mounts.

Of course, actually performing unlimited mounts might be a good way to bump into other kernel limitations that aren't quite as obvious as the number of loopback devices. A better solution is as follows:

Create a file system using FUSE that supports the desired AppDisk file system types, and mount it to the proper point in the system. (I believe I suggested /apps.) When you want to mount a new AppDisk, you simply call the API for the FUSE file system, and the contents of the AppDisk will magically appear inside a sub-directory of /apps.

This scheme is really just a VFS inside a VFS, but it has the advantage of limiting the resources allocated by the kernel. Instead, all the resources would be allocated in userland where they can be easily tracked, protected, and paged.

Implementing the DBFS

Of all the concepts presented in the original article, the DBFS is probably the least developed in the OSS community. Because of this, I have spent considerable time attempting to derive quick methods of implementing this system. The three schemes I've come up with are as follows:

Direct Implementation
MySQL & FUSE
Berkeley DB & FUSE

Direct Implementation

Directly implementing a custom disk layout is the most desirable method of achieving a DBFS. It would have the advantage of being supported by a true kernel module, it would be easy to tailor to emerging needs, could use block devices directly, and would be much easier to tune for performance. It would also be possible to make such a DBFS a bootable drive, since the developer has direct control over the file system's structure.

The disadvantage is that it would take quite a bit of time to implement. Not only would the data layout structure have to be developed, but a record layout structure and indexing scheme would also have to be worked out. These concepts are well known, but would take time to complete.

MySQL & FUSE

I realize that I just debunked the concept of using an SQL Server for the DBFS, but I would be remiss in my duties if I didn't at least mention the possibility.

The absolute fastest method of getting the DBFS online is to use a loopback file system (i.e. a new view on an existing file structure) to attach meta-data to existing files. The concept is quite simple:

Setup a MySQL database to store the meta-data.
Create a FUSE file system that translates VFS calls to the appropriate file on an existing file system. (See the fusexmp example in the FUSE source code for an example of such a file system.)
Present the user and programs with a DBFS view on the data.
Profit!

It really is that simple. Unfortunately, there are quite a few drawbacks to this scheme. For one, the database server is vulnerable to crashing, corruption, rouge queries, security flaws, and a host of other issues. For another, the stability of the DBFS is vulnerable to the mapped file system being changed independent of the DBFS view. Finally, this scheme would be extremely wasteful of the system resources and could potentially interfere with other MySQL installations.

Berkeley DB & FUSE

Probably the best compromise between the two concepts presented above is the use of a high performance embedded database for storing both the meta-data and the file's binary data. Such a database could handle all of the disk layout and logging operations, as well as the indexing and querying.

This is where Berkeley DB comes in. Berkeley DB is a simple yet powerful database designed to store strings of arbitrary binary data. It does this by allowing one table of keys and values in each database. A key is a binary string and a value is a binary string. That's it. There's no concept of columns, relations, data types, referential integrity, or any other fancy database concepts. If a structure is desired, it is expected that the program will impose it upon the value portion of the record. As a result, the value portion of a BDB record is often a C Struct.

Other features of interest are BDB's ability to store multiple databases in a single file, support for indexing on parts of a record's values, automatic memory mapping, support for sequences, transactional safety, support for large records (up to 4 gigabytes in most systems) and the ability to retrieve/store only parts of a record value. The latter two points are especially useful, as they allow for a Berkeley database to be used to store all the data for a file. In effect, BDB can become the file system!

The first feature is also of interest, because it means that BDB can be used directly on a block device if so desired. While it may be tempting to jump directly to using a block device, it is important to consider that placing the file on the root file system allows for disk space to be allocated to either the root or DBFS as needed. If you make the DBFS a separate partition, then you become bound by the disk sizes you chose.

What Berkeley DB does not provide is a fancy network server, user management, a query language, or any other cruft that might get in the way of a DBFS implementation.

Implementing a DBFS under Berkeley DB

You didn't think I'd leave you with only a quick description of the concept, did you? Believe it or not, I've been hard at work attempting to work out the details of creating a DBFS in BDB. Precisely three databases would be required for basic file system support. (Keep in mind that a database in BDB is analogous to an SQL Table.)

The following are what the value structures might look like. Note that all keys are expected to be long integers.

struct {
long created;     //Date
long modified;    //Date
long accessed;    //Date
char* name;       //Variable length
} files;            //The key is a sequential integer

struct {
long modified;    //Date
char type;        //0 for string, 1 for data, the rest reserved
char* name;       //Variable length; indexed!
char* value;      //file_data_id if type == 1, else variable string
} meta_data;        //The key is a sequential integer

struct {
char* data;       //Whatever the heck we want to put in here.
} file_data;        //The key is a sequential integer

In looking at the structure above, you may begin to wonder where exactly the file data would be stored. The answer is, in the file_data database! Despite the high importance placed on the data inside a file, the data is really nothing more than an another attribute or piece of meta-data. Files would have a standardized meta_data record called ".data" or something similar.

Adding labels (which can then be presented as directories for backward compatibility) can be done with the following database:

struct {
char* name;
long[] file_ids;    //A list of files attached to this label
} label;              //The key is a sequential integer

Building a Query Engine

Obviously one of the biggest reasons for creating a DBFS is to allow for advanced queries. To some degree the structure presented improves upon searching. If you were looking for a file created within the last 30 days, the DBFS could do a table scan to find files with the requested date.

Assuming that the "files" database key was a "long" integer, and that the average file name size was 50 bytes, the DBFS would have to search through 7.8 megabytes of data to find all the files created within the last 30 days. Since the data is stored in a table instead of scattered across INodes on disk, a burst read from the hard drive could read through the database in a few seconds or less! And that is assuming that none of the data is already memory mapped and cached in RAM! (Which is a very likely scenario.)

Unfortunately, a database scan does nothing to help the issue of searching file contents. To fix this, we need an index. Thankfully, it is quite easy to create one. All we need is a database where the key is a string, and the value is an array of file ids. e.g.:


struct {
long[] file_ids;
} word_index;        //Key is a String such as "hello\0"

Generally it makes the most sense to store only the lower case variants of each word to save space and eliminate issues with case sensitivity. This still presents a few issues with internationalization (the rules for lower casing a word differ from language to language), but that's not an issue I'm quite ready to tackle.

The end result of this is that a query engine built on top of this index could quickly find a list of file for each keyword. By creating a combined list (or), subtracting one list from another (not), or creating a list of only duplicate matches (and), boolean searches can be achieved.

The Query API

After playing around with FUSE for a little while, I realized how a simple and backward compatible query engine might be achieved. Consider a URL for a website. It usually consists of a protocol (e.g. http://), a host (e.g. myplace.com), a path (e.g. /myfile.cgi), and a query string (e.g. ?myfield=value). In the case of a file system, the protocol and host are both known, and the path is supplied by the user. However, no concept of a query string exists.

What if we were to virtually provide access to files beneath a given file path? For example, let's say we had the following file:

/myfolder/myemail.msg

Let's say that the file had the attributes "subject", "to", and "from". We could access the contents of each of these attributes by opening the following paths for read-only access:

/myfolder/myemail.msg/#subject
/myfolder/myemail.msg/#to
/myfolder/myemail.msg/#from

Note that the # symbol is used to denote access to an attribute. Similarly, we could write the attributes by opening the file for writing. Futhermore, we could get a list of attribute names by opening the following file for read only access:

/myfolder/myemail.msg/*list

These files would never actually appear in a file list because they are entirely virtual constructs. The file system driver would parse the path, find the file that terminates the path, then return a handle to a reader/writer for the attribute or list.

This scheme can be further extended for general queries. The following query finds all files with the word "batcave" in them:

/?query=Batcave

Since special characters are not allowed in an indexed word, the standard GMail syntax of using [attribute name]:[keyword] can be used to search only specific attributes. Combine this with a feature for querying only specific labels, and you could craft a very advanced query such as one that looks for email from Bob:

/myfolder/?query=from:Bob

To obtain the information, the query engine would look up the label "myfolder", and find all files associated with it, as well as sub-labels and files. It could then search through the meta-data for each file looking for the attribute "from". When the "from" attribute is found, the engine would then attempt to perform a substring match against the value of the attribute. This process can be accelerated by adding the attribute values to the index database we discussed above.

Advancing the State of Applications

One of the seeming disadvantages to the AppDisk/DBFS scheme proposed in the original article was the idea that program settings would be lost when the application was deleted. While this is a valid concern, I don't think it is as big of a concern as it might seem.

A DBFS not only advances the user's ability to interact with a file system, it should advance the ability of applications as well. The primary reason why users don't want to lose program settings after removing an application is because of the databases that application has stored. For example, my email program may have my entire email database in its settings. Or my web browser is holding all of my web bookmarks. Why should these programs hold our data hostage? Isn't the Unix philosophy "everything is a file"?

The reason why programs hold our data hostage is because they need to store it in special database files for performance and usability reasons. For example, it would be far too slow to parse out a directory full of emails just to show you a list of your inbox. Such an email program would easily be eclipsed by a email program utilizing an indexed database. Unless the former program had a DBFS backing it, that is!

The syntax still needs to be worked out, but imagine if you could open a query underneath a label that asked for a textual list of files with a given attribute and the value of that attribute! The email program could ask for the list of files with the MIME attribute of "text/email", and a list of the "to", "from", and "subject" attributes! The resulting file stream might look something like this:

myemail.msg, Bob, Me, How are things going\, Bob?

Note that the comma in the subject had to be escaped. Programs who read from these files will need to be aware of this issue, or perhaps would be allowed to switch the delimiter to a character of their choosing.

As a bonus, this information can be processed by traditional command line tools! Nothing stops you from SSHing into your machine, requesting a list of emails, then using grep, sed, cut and other Unix tools to process the data and generate a report!

Long term it would be valuable if all programs adopted this scheme. Your Web Browser bookmarks would be more accessible than ever, your address book could be nothing more than a bunch of VCF files on disk, and your calendar could consist of a number of event files. Not to mention that programs would have an easier time sharing this information! I for one can't wait for the day when Mozilla, Opera, KHTML, and other web browsers share all my bookmarks.

Links:

FUSE
Berkeley DB
Micro Kernel
SSHFS
GMailFS

The author can be reached at akaimbatman@gmail.com. Watch your step when entering the Batcave!

Linux Desktop Distribution of the Future Follow-up Part 1

2005-07-07T20:55:00.000-07:00

Category: Commentary

Part 1: Clearing Up Misconceptions
Part 2: Refining the Ideas

Originally this week was slated for an article unrelated to Linux, yet jam packed with useful and interesting information. Sadly, that article will have to wait. The response I received to my four part series "The Linux Desktop Distribution of the Future" was overwhelming, and demands a follow up. As a result, this article will focus on answering some of the common questions and criticisms leveled against the previous series.

Talkback

The following is a list of everyone I know who answered the series outside of the blog comments. Feel free to dig through these to get a feel for both the criticisms and the accolades that were put forth.

Slashdot.org Story
OSNews Story
Linux Today Story
Mark R. Hinkle's Blog (Editor of Linux World)
Brains Factory
Phil Crissman's Blog
Eric's "Extreme Boredom" Blog
Willisburg.org
House of Zeus

I do know of a few more, but my Russian is rather poor and I don't speak most Germanic and Arabic languages.

Direct Responses

There are two articles I am aware of that were published as direct responses to the series. The links and my responses are listed below:

EMerge Random

- or -

Does that Toaster Come With a Manual?

Dev/URandom's premise for responding is very odd at best.. His idea is that Linux is ready for the Desktop today, just as long as the user reads the manual first. He then attempts to drill home the concept by using expanded text like this: "What I answer to my John Doe's that want to install Linux is: read documentation first. Please understand to the last word: f i r s t."

The problem is that his argument does not hold water. Users don't read the manual before they drive a car, microwave their food, use a refrigerator, or even use a Desktop computer. In fact, Mac OS X comes with no manual to speak of! If Linux requires that the user read the manual before using the interface, it has already failed.

Thankfully most respondents set him straight. He has now changed his tune and moved on to a slightly more intelligent premise.

Jeff Williams

Mr. Williams' rebuttal was much more even handed than Dev/URandom's, and was a pleasure to read. Unfortunately, Mr. Williams seems to misunderstand my post in a few places.

In Part 2 of his response he refers to "installation" software for the disk images, and pretends that the disk images are just a reformatting of existing package systems. What Mr. Williams does not understand is that the AppDisks are never installed. This is a very important thing to understand. The AppDisks are mounted in place, then run directly out of the image. When the application exits, the image is unmounted and thus leaves the system. This is worlds different than the packaging systems of today.

In Part 3 of his response he refers to the DBFS as "more conjecture than something really implementable." Unfortuantely, he does not expand on this, so it's difficult to formulate a response. All I can say is: Yes, it is doable. Most everyone out there could do it with a standard database server. I'm suggesting doing it at the file system level. In this case, CompSci is on my side.

In the section entitled "Core system libraries," Mr. Williams attempts to agree and disagree at the same time. On one hand he says that I have identified the key issue with Linux (a lack of standardized APIs) then states that Fedora and Debian are standards. What I think Mr. Williams is missing is that the exact APIs can be modified at install time. The user can choose to install KDE, or he can choose not to. He can choose to install the SAMBA components, or he can choose not to. That choice is a powerful advantage for workstations and servers, but a very poor concept for a Desktop system where the user wishes to make his applications Just Work(TM).

In the section called "Documents", Mr. Williams argues that I have contradicted myself about special directories. However, he fails to grasp that the /Documents directory is not a special folder. It is, instead, a replacement for the user's Home directory. Inside it, the system is normalized to allow the user easy access to his documents, applications, and other functionality. This is different from GNOME and KDE which create various bits of files and folders (both hidden and visible) that have special meaning inside the system.

Finally, under "Part 4: Desktop", Mr. Williams fails to understand the concept of the saved query. His belief is that the results of the query would be saved. That was not my suggestion. My suggestion was intended to be to save the query itself, then rerun it on demand. Since the file browser would constantly be running queries, the saved query file would merely pass its contents to the file browser for execution. The results would then be displayed as if it were a "special" folder.

Now on to the more general points.

A DBFS is not an SQL Server!

One concept that seemed particularly difficult for people to accept was the idea that a file system could both serve files and be a database at the same time. From the responses, everyone seems to have gotten the idea that a heavyweight SQL Server would be required to support the file system, and that the linkage would be weak at best. If you're one of those people, place your mind at ease. I suggested no such idea.

To understand why a DBFS does not need a separate server, one must first understand that a file system is a database. As databases go, most are quite poor, but they are still databases. Despite the fact that most commercial databases maintain relationships and provide client/server abilities, all that is required to have a database is to store data in a structured fashion. By that definition, even a comma delimited file is a database!

Speaking of comma delimited file, the original computer databases were not far from these. Data was stored in records of fixed lengths, and retrievals were done by scanning the file. Columns existed only inside the program's definitions. i.e. A program might say "Characters 0-10 are the name, while characters 11-25 are the address". This scheme worked well enough, but wasn't particularly fast if you needed only a few records from the data.

Indexes were created as a solution to this issue. Indexes are really nothing more than a data structure that is faster to navigate than a full database scan. When these indexes were added to a flat database, the resulting database was referred to as an "ISAM" file. ISAM files are still used in many systems today due to their inherent simplicity.

Indexes can come in many shapes, sizes, and forms. The only purpose of an Index is to improve access to a particular piece of data. Now think for a moment. When you access a file on your hard drive, does your machine spin through every file on the system looking for the correct one? No! It provides you with a directory structure to navigate that allows you to pinpoint the correct file. Believe it or not, by navigating the directory structure, you've just followed through an index.

Fast forwarding to today, the underlying database technology has seen almost no change what so ever. Database servers build relational theory and client/server technology on top of indexes and data access, but offer little more at a lower level other than abandoning the fixed record sizes for indexed record/column locations.

In the context of my original article, my suggestions were nothing more than reorganizing the data structures that already exist on disk drives today. This is to provide a query method that does not require a jaunt through the directory structure. There is nothing incompatible about this concept, and it can already be seen in the myriad of file systems that are available today. NTFS is probably the closest file system to what I am suggesting, but BeFS and HFS+ both provide highly advanced data structures and indexing in their designs. Isn't it interesting that Linux supports all of those file systems just fine?

Yes, disk creation and repair tools would be required for this new file system. However, the same is true for every file system ever added to Linux. When you run FSCK, you're not running a generic program, you're running a wrapper around a file system specific program.

Have you tried Distro XYZ?

This is quickly becoming the most annoying question in the history of mankind. It doesn't matter *which* distro you've tried or haven't tried. They all run on similar principles. No one distro magically solves all the issues facing Linux today. Sure, some have strengths over other distros, but they also have weaknesses. As a result, you find yourself in a conversation like this:

Me: I tried RedHat, but XYZ was giving me headaches.
FanBoy1: You should try Gentoo, it fixes all your problems!
Me: I tried Gentoo, but problem ABC was a show-stopper for me.
FanBoy2: Gentoo is crud! Use SuSE!
Me: I tried SuSE, but it didn't support 123.
FanBoy3: That's because you should have been using Debian all along!
Me: I tried Debian, but it broke.
FanBoy4: That's because Debian is out of date. Ubuntu is the future!
Me: Ubuntu still has issue Z.
FanBoy5: I understand all your issues with Ubuntu. I ran into them myself, so I switched distros. You should really try RedHat!

And so this crazy circle completes.

For the record, I have tried nearly all mainstream Linux distros. You can find some very nice reviews of some of them in the history of this blog. (Originally posted in my Slashdot Journal.) No, I have not tried Ubuntu yet. I'm still waiting for my CDs to arrive due to this blog keeping me too busy to download and burn CDs.

Installation != Ease of Use

An interesting fallacy I've seen as of late is the idea that an easy installation somehow equates to ease of use. Let me put this to rest. The only reason why the user cares about how easy it is to install Linux is because Linux doesn't come bundled with hardware. Windows and Mac OS X don't have to worry, because they are already installed for the user.

Most Linux distros today do an excellent job of providing a user friendly installer. Which is great if the user has to install his OS. For long term usage however, the user doesn't really care about the installer. He cares about things like using whatever application he wants, having document compatibility, playing games, keeping his work organized, and above all else having his machine work for him instead of against him.

Far too often Linux works against its users, because it wants someone more experienced at the helm. There's nothing wrong with that as long as Linux is targeted at the workstation and server markets. But if you really want to see Linux adopted on the home or work desktop, then you need to consider how to keep Linux from fighting its users.

Shortcuts vs. Labeling

Another item that seems to have confused readers was the issue of keeping shortcuts off the desktop. I understand the confusion, so allow me to clarify: The desktop I'm proposing has no shortcuts in the traditional sense. Instead, the labeling system is used to provide a desktop label. Anything linked to the desktop label will show up on the user's desktop. This works because files can be linked to multiple labels. Thus if I have a file under the "Text Documents" label and add it to the "Desktop" label, the file will appear in both places.

Dumbed Down for the "Smart" People

An interesting criticism of my article was that the concept was "dumbed down" for the idiot savant user. If you'll forgive me, I find this to be an offensive idea. "Dumbing Down" implies that something is reduced in functionality and features until only the absolute minimum is left. An example of this is the arcade games designed for three year olds that only have one button and no joystick.

The concepts I introduced were far more sophisticated than those used today. The current file systems, for example, are quite dumb. (Except for ReiserFS. I considered the possibility of Reiser solving the DBFS need, but I found its meta-data support to be incomplete.) The DBFS I proposed would actually be quite smart. The interface I suggested contains all existing user interface component, adds some functionality, and has far more potential for the future.

The is nothing "dumb" about the suggested changes. All they do is Keep It Simple Stupid.

AppDisks vs. AppFolders

A very important distinction that seems to have slipped past many is the fact that I suggested the use of AppDisks, not AppFolders. These AppDisks are *similar* to AppFolders, save for the fact that they wrap a complete unix hierarchy into a mountable disk image.

This means that /bin, /lib, /share, and other Unix style directories still exist. For example, shared libs can be stashed inside the /lib directory, in direct opposition to the way that Mac OS X AppFolders work.

Linux Applications are so easy to install, they don't work

A common response to my article was to gloss over the issues with the packaging system and claim that there's nothing easier. Well, yes, it's quite easy to click and install from a manager like Synaptic just as long as the package is available. Many programs that users wish to run have no package, out of date packages, or the repository has moved on to a newer version, thus making their Linux system useless. In the latter case, the solution is to upgrade your system. Yet a common complaint leveled against Microsoft is that Microsoft forces users to upgrade!

In addition, there is no way to fully test a package repository. Since every package modifies the base system, the only way to prove that a package will work is to test it against every possible package configuration available! In case you're wondering, the math for that is P * P, where P is the number of packages available. A mere 100 packages could potentially result in 10,000 available configurations! That's a lot of potential for breakage! Now consider that most distros today have thousands of packages under their care, and the number is not declining.

Minor Correction: Reader Bradley Momberger has correctly pointed out that my math was a little screwy on this one. The correct forumla for the number of combinations is 2^P, which is actually quite a bit worse. 100 packages yields 1.26e30 possible combinations!

In the AppDisk system I've proposed, the application is merely a visitor to the system. It lives in its own directories and its own environment, separate from the rest of the system. Thus there is precisely P possible configurations to test. And since the one to one relationship exists, it is quite feasible to place the difficulty of testing the application on the developer.

This is all Sci-Fi Technology!

Perhaps the most amusing criticism is the suggestion that the ideas presented are "Sci-Fi" technology that couldn't possibly exist without multi-million dollar backing and years of development.

This argument, I'm afraid, ignores the fact that I'm building on top of a great deal of existing work. i.e. Standing on the shoulders of giants as it were. Take the example of applications as disk images. While I invented the concept independently, I can't take credit for being the first to deal in the concept. That honor goes to the Klik project and their CMG file technology. Granted, their CMG technology is still very immature, but it is here and working today.

Another example is the concept of creating a running system with the applications outside the /usr directory. GoboLinux has gone to great pains to experiment with alternate file system layouts. Yet another example is Rox Filer/Rox Desktop, the pioneering project behind the use of AppFolders in Linux. (GNUStep had AppFolders first, I believe, but the applications are restricted by the OpenSTEP Desktop.)

Look at the work being done by the DBFS and Beagle projects for examples of Database File System technology.

The only thing I have attempted to do is make suggestions about how disparate technologies may be pulled together to make a cohesive whole. Make no mistake. I am not suggesting a Linux desktop of the future, I am suggesting the Linux desktop of the future. Whether I play a part in it or not is irrelevant. It is already happening. The question is: Will we beat the competition or will we continue to play follow-the-leader?

In the next article, I will attempt to refine some of the concepts presented based on the feedback I've gotten. So tune in next time, same Bat Time, same Bat Channel!

Go to Part 2: Refining the Ideas >>

Links:

NTFS
BeFS
HFS+
ReiserFS
CMG Files
GoboLinux
Rox Desktop
DBFS
Beagle

Questions? Comments? Business Ideas? Hate Mail? You can send it all to akaimbatman@gmail.com. (Except for the hate mail. That belongs in /dev/null.)

The Linux Desktop Distribution of the Future Part 4

2005-07-01T20:09:00.000-07:00

Category: Conceptual Design

This article is part of a four part series intended to provide some thought into how a future Linux Desktop might work. It is not intended to be a comprehensive essay, although all the concepts presented here are considered "doable" by the author.

Part 1: Linux and the Desktop Today
Part 2: Applications
Part 3: File Management
Part 4: The Desktop Interface

Part 4: The Desktop Interface

In the final edition of this series, I'll tie together the technologies previously discussed, and explain how they might coexist in an easy to use desktop environment.

The Layout

When a user attempts to be productive with his computer, there are usually only a few types of files he's interested in: Application and Documents. The Applications are the programs he uses to work with his documents. Just about everything else that a user does is intended to support this functionality. For example, the user has no direct need for a trash can. Rather, the trash is an abstraction that allows him a second chance before permanently deleting a set of files.

As a result, the only things that should be on a user's desktop are:

1. An icon to access his applications.
2. An icon to access his documents.
3. An icon to access his trash.
4. Any special mounts such as CDROMs, Network Drives, Cameras, USB Storage Devices, etc.

Everything else should be kept off the desktop. In particular, it is rather important for the system to NOT have desktop shortcuts in order to prevent the common glut of special offers and installers.

For the purposes of easy to access files, it is in the user's interest to allow selected files to appear on the Desktop. In the proposed interface, the Desktop would be merely a label used by the system to identify which files should appear. As a result, the right click menu and/or toolbars can provide the user with the option to add or remove the file from the Desktop. The key difference between how this would work in a DBFS vs. a regular system, is that the file is never moved in a DBFS. If the file is already organized, it will not have to move to appear on the Desktop. Rather, the file simply has the Desktop label added. Removing that label would have no effect on the file other than to make it disappear from the desktop area.

The two other key items on the Desktop are the search box and the task bar.

Applications

Clicking on the Applications icon pops up a file browser displaying the contents of the Applications label. This label can theoretically contain anything, including sub-labels, but will hold all applications by default. In the filters section we'll discuss a method by which applications can automatically end up here.

Documents

The Documents icon shows a file browser of the root of the label tree. Uncategorized files show up in the root, while any file with at least one label can be found underneath the selected label(s). The Applications and Trash labels are automatically weeded out to keep the user from getting confused. Note that what is displayed is not the true root of the filesystem. Rather it is a virtual root consisting only of files and labels that have no parent. No true hierarchy exists in a database file system, so the label and file information is used to compute one.

A special "label" called "Users" should show up under here, with a set of sub-labels for all users of the system. Everything beneath the user's name is part of their DBFS files and meta-data.

Trash

As mentioned previously, files intended for deletion are simply labeled with the "Trash" label. Unlike other labels, however, any file with the Trash label should be automatically hidden from the file browser unless the Trash is being explicitly stored. Should the user decide to rescue the file from the trash, the system merely needs to remove the Trash label from the file in order to restore its full set of labels and attributes. Should the user decide to empty the trash, only then is the file actually deleted.

Note that at no time is the file actually moved or modified. The "move" to the trash can is merely an illusion designed to provide a mechanism similar to the trash can in today's OSes, but far more robust thanks to the ability to leave the file in place.

Search Box

The search box in the upper right hand corner is a search like Apple's Spotlight and a command line box rolled into one. If the box detects that you have typed an absolute path, it will bring up a file browser window regardless of the fact that it's outside of the DBFS. This provides the user with a way to browse the system files if he so desires without resorting to a terminal program. The search box can also have a URL or URI entered to directly access an Internet site or shared network drive.

In all other circumstances, the search box will automatically search all files on the DBFS, sifting through indexes and meta-data to find any and all matches. Since this query can effectively be pushed down to the DBFS, the concept is that it would be tremendously fast because of the DBFS indexes.

Future enhancements to the search box might allow for more complex functionality such as quick commands to run programs, web searching, program interfaces, dictionary lookups, and other features similar to the way that Google provides a combined search and command line today.

Another important feature to include in the Search, is a method by which the search can be saved as a pseudo-folder. In reality the search should be nothing more than a file on disk that opens to the search results window (thus allowing the file to accept regular labeling and meta-data additions), but the icon for the saved search would make it appear to the user that the search is a special type of folder.

Taskbar/Dock

It is generally agreed by all modern UI experts that there must be a sensible method for the user to view and access all open program windows. The most popular of these methods has long been the Windows Taskbar, which shows a button for each open Window. To date, nothing has been found wrong with this interface other than potential usability issues that result when the user has an overcrowded taskbar. Still, it's a familiar interface for most users.

Similarly, the Dock is very well known to users of Mac OS X. The Dock interface provides a solution to overcrowding by automatically scaling icons so that all the icons can be seen on the screen at once. The icons are never too small, because they regrow as the user moves the mouse over them. This allows the user to manage hundreds of programs with the flick of a mouse over the Dock.

So which is better? The answer is: Whichever one you prefer. Some people prefer the Windows method and some people prefer the Mac method. As a result, this is one of the few places where giving the user an option makes a big difference. i.e. Let the user choose which one he wants. They're similar enough to implement via the same methods, and this desktop has no extra baggage to differentiate either design at an API level.

Filters

Anyone who has ever used email can tell you that the ability to setup rules on incoming email is an invaluable method for organizing and managing your email. GMail users can tell you that when combined with Labels instead of folders, those rules (called Filters in the GMail system) become even more valuable than before. For example, in GMail you can automatically organize your email under the proper labels, but still see the email in your inbox prior to archiving it. This saves a great deal of time, as the user does not have to check each email folder individually.

When applied to downloaded files, this raises an interesting question. Why don't web browsers have rules to automatically organize downloaded files? Surprisingly, the answer is the same as with email rules for folders. By setting up a rule, an email bypasses any sort of holding area (the inbox in most cases) where the user can be easily informed of its existence. Users have enough trouble attempting to find their downloaded files today without running into problems with files automatically moving to God knows where.

Yet just like with email, the problem goes away when labels are introduced. A file can be automatically organized under a label without removing it from a "downloaded" label or the Desktop. Smart software could even auto-manage the user's downloads by automatically removing a file that's linked to another label after a given period of time.

The best implementation of this is to push the rules down to the filesystem level. When a new file is created, the DBFS will check the filters and if any apply, use them to automatically apply labels to the file. For example, if I create a new SXW document, I could have a rule that would automatically place the document under the "Word Processing" label. Nothing prevents me from later adding a project-specific label, but with the filter in place I know that I can always find all of my SXW documents in one place. Similarly, I can set a filter stating that all MPG files with the words "Star Trek" in the name should be placed under the "Videos" and "Star Trek" labels.

The power that such a system as this gives the user cannot be overstated. While search can reduce the amount of time a user spends looking for a file, filters all but make organization unnecessary. If a type of file isn't properly trapped by the system, the user can always add another filter.

File Save Window

Since this new Desktop and Filesystem paradigm does away with the previous concept of directories, it becomes important that the File Save dialogs be updated to reflect this shift. While the old boxes will work fine for a time (as they will see the labels as if they were directories, and the DBFS will automatically assign the selected label), a more robust solution would be a save dialog that shows you the labels in the system, and allows you to build a list of them in addition to giving the file a name and file type.

This could be implemented for many current applications by modifying the existing GTK and QT file choosers. Unfortunately, the remaining applications would need to be modified to fully support the new Labeling system. Not to worry, however! Even with applications that are unable to break out of the legacy file choosers, file system filters will still allow users to automatically organize their system despite a given program's failings.

Filter Manager/Control Panel/Network Browser

One of the greatest crimes against good UI design was the attempt to move "common" functionality directly into the Desktop metaphor. Such functionality only served to confuse users and complicate the interface. Yet integrating such functionality was almost required in order to keep the user from having to sort through dozens of menu choices before finding the actual system options.

Thanks to the Applications as Files scheme, there is no need for the user to have to put up with this any longer. Functionality such as the Control Panel, Network Browser, and other system features should show up as "just another application". If the user is savvy enough to know they need these apps, the user is savvy enough to look in the Applications label to find them.

Which isn't to say that other programs might not invoke them. The key is that the interface is only provided when it makes sense, and never at any other time. The upshot to this design is that these interfaces can actually be upgraded independent from system components. Which is a good thing when you consider that they are nothing more than GUIs that assist the user in managing system configuration. If a user really misses having such options available from the Desktop, he can simply add the Desktop label to the applications and watch happily as they integrate right into his desktop!

Notably Absent

There are a few things that are noticeably absent from the proposed Desktop concept. For one, there is no "Start Button" or equivalent. This is intentional. The Start Button was always a confusing metaphor that only existed to cover over the inability of the Desktop to function in a cohesive fashion.

Another thing that you'll note is missing is the concept of a shortcut. This is intentional. Shortcuts are dangerous metaphors because they don't keep in sync with their target. They were created as a method of covering over missing functionality in the Desktop (e.g. Listing application shortcuts under the Start Button) as well as covering over the missing ability of the file system to non-rigidly link a file. Consider for example, what happens if a file open operation is done on a shortcut? Should a handle to the target file be returned, or should a handle to the contents of the shortcut file be returned? If the answer is the former, then how does the system edit the shortcuts? Is a special FNCTL call needed?

As you can see, leaving these concepts out of the desktop only helps improve the situation. The only exception to the no-shortcuts rule is the Dock interface. If the user is using the Dock interface, then he will probably have shortcuts to commonly used programs located there. A possible solution to this issue is to create a "Dock" label. Any applications carrying that label would automatically appear on the Dock regardless of whether they are currently running or not.

Issues Remaining

As with all high level discussions, this article still leaves various issues unanswered. For example, how do users share applications? Are all applications available to everyone all of the time, or is there a method by which system wide applications can exist independent from user owned applications? Such issues such as this are easily solvable if given enough thought. The key is to be cognizant of the new paradigm that these changes bring to the table.

The author can be reached at akaimbatman@gmail.com.

The Linux Desktop Distribution of the Future Part 3

2005-06-23T20:39:00.000-07:00

A partition for the core system libraries, root files, and /usr files.
A DBFS partition for the user's applications and documents.

Core System Libraries

One of the most important things that a desktop user needs is to get out of the business of maintaining system files. Such maintenance was always problematic for Windows users and even gained the title of "DLL Hell". Microsoft eventually solved this issue through several OS changes, not the least of which was encouraging application writers to keep their DLLs to their own program directories. Most Windows programs today install into a folder in the "Program Files" directory with no extraneous files in the System32 directory.

Linux, on the other hand, took the approach of package management. No real standard existed for the base system, but rather the core libraries could be updated at will. This lead to the situation where a given distro version may mean a different set of available APIs under each installation. Standards such as LSB have been proposed to help alleviate this situation, but such efforts have mostly focused on providing a minimum of low level APIs. What is necessary in a Desktop focused distro is that the developer be able to count on a specific set of APIs for a given system level.

For example, let's create a mythical desktop Linux OS called DeskLin. For version 5.0, DeskLin publishes APIs for GTK 2.1 and QT 3.2. Thus all programs based on GTK 2.1 or lower and Q 3.2 and lower should work. If a developer wants to create a program that uses the FLTK toolkit instead, he'll need to package the FLTK shared objects in the lib folder of his application bundle. Now let's say that DeskLin Inc. notices a large number of applications using the FLTK APIs. To reduce waste, they may chose to include FLTK as a core API in DeskLin 6.0. This then allows software application developers to take advantage of these APIs as long as they target version 6.0 and up of the DeskLin operating system distribution. The application developer, however, can still target version 5.0 by including the FLTK libraries.

The end result of this process is that control of the system APIs is taken away from the user. While many Linux purists would argue against such a step, it's important to note that I am not advocating taking this step for all distributions. In workstation and server environments it can be critically important that the user maintain complete control over his system. Only Desktop-oriented distributions that are looking to target home users should break off and take these steps.

Of course, security updates will still be an issue. As a result, it makes sense for a Desktop distribution to carry an installer mechanism such as Autopackage. This installer mechanism would allow for patches to be applied to the system quickly and easily. If possible, such patches should be automated. Beyond that, the core libraries should be hidden from the user and made read-only.

Documents

Differently from system libraries, users DO want to manage their documents and are always happy when they are given more power to do so. The best solution for users is to move their documents and only their documents into a database file system. In many other OSes this might be a problem, as there only exists one filesystem tree. In Linux we can get away with much more.

Under Linux we can create a new VFS module that handles a DBFS partition independent from the system files partition. The DBFS can then be seamlessly integrated by mounting it onto a standardized subdirectory such as /Documents. To a terminal and classic Linux programs, the DBFS Labels would look like normal directories. Files outside of the /Documents folder would be non-writable for normal users. Queries to the file system can be performed through a special /proc or /dev interface, allowing the Desktop to quickly search for the exact files the user is looking for.

This arrangement does pose a few problems, however. In a database file system, it is possible for the same file name to occur more than once. Possibly even under the same label. As a result, it is very important for programs to start using the INode number for the file instead of the abstract path. Two solutions immediately come to mind for dealing with "Classical" software programs:

Allow for the final component of the path to be the INode number. This could be achieved with special file names such as "#1234" or "@1234". Such names are not normally used in an everyday Unix system, and are thus unlikely to be invoked by accident. These names can be passed to existing software programs to ensure that the correct file is selected. There is still an issue if the program displays the filename, however.
Take a page from the Windows VFAT scheme and display duplicates under the same label with special filename extensions. For example, two files named "Bill.doc" could become "Bill #1.doc" and "Bill #2.doc". The only downside is that the user may become confused at the numbering scheme. Especially if he attempts to rename the file to include the special addition.

Configuration Files

The one type of file that I haven't yet addressed is program configuration files. The reason for this is that I currently have no "good" place for them to go. The Windows solution of using a central registry is certainly not a bad one (although definitely not a good implementation), but adds one more complex abstraction for the user to deal with. In addition, it is far less feasible to force Linux programs to make the switch to a registry solution than it was for Windows programs. (Central control does have its advantages.)

The best solution I can come up with at the moment is to hook into the Unix "standard" for configuration files in order to create a virtual registry in the file system. The idea is as follows:

When the DBFS detects a file or directory created with the "." prefix (a naming convention that hides a file in Unix), it immediately traces back the creating program to its on-disk disk image. Instead of creating the file, the DBFS makes it a binary meta-data attachment to the program image. The resulting psuedo-file is then indexed by the DBFS so that a regedit-like management program can quickly retrieve a list of all programs with such configuration files.

The downsides to this scheme are as follows:

The application settings are permanently attached to the program. If the software application disk image is deleted, the settings will go with it. Many users would see this as an advantage, but it does depend on the program. Some larger programs are removed and later reinstalled by users due to disk space considerations. Gaming users in particular might not be happy if their Doom III saved game was lost. (Although the obvious solution to that situation is to encourage the vendor to change the saved game to be a regular document that is associated with the application.)
Inter-program communication through configuration files is made more difficult. For example, Opera would have to support the necessary DBFS APIs to directly access the meta-data under which pre-existing bookmarks are stored in FireFox. In a traditional Unix system, Opera could just open the "."-prefixed sub-directory in the user's home folder.

DBFS Structure

While the technical details of how the DBFS stores files and meta-data is for the most part irrelevant, I'm going to quickly go over the structure to eliminate questions concerning what a DBFS can and can't do.

A DBFS as envisioned in this article would forgo the traditional storage of INodes and Directories. Instead, the data about the filesystem would look more like an SQL database. An entry would exist in the database for each file on disk, with keyed linkages to meta-data. The meta-data list would be like a table with three columns: The name of the meta-data, the type of the meta-data, and a link to the value in another table. Tables would exist for String, Integer (64 bit?), and Data Block types.

The Data Block type would be a table that would hold a list of file system blocks used by the specific piece of meta-data. All files would have a piece of meta-data of this type that would identify the file contents. (Likely named something witty like "data".) Strictly speaking though, Data Block meta-data other than the file contents may be attached. For example, the application icon may be stored in a meta-data value called "icon", or a thumbnail of a photograph may be stored under "thumbnail". In fact, the potential for storing pre-calculated binary meta-data is limited only by disk space and your imagination. Even plain old text can easily be extracted from meta-data already in a file, and stored in the file system for easy access.

Each of the "tables" in the DBFS would be appropriately indexed to provide for the fastest lookup times possible. It's likely that such indexes might also be created for the contents of a file. These indexes would be what would make fast file searches/queries possible.

Tune in next week for the final installment, where I tie all of these features together into an easy to use interface!

Part 4: The Desktop Interface

Links:
LSB
Autopackage

The Linux Desktop Distribution of the Future Part 2

2005-06-15T20:38:00.000-07:00

The Linux Desktop Distribution of the Future Part 1

2005-06-15T20:05:00.000-07:00

Installing Applications is complicated
Directory structures can be confusing to navigate
Interface is confusing and inconsistent
Steep learning curve required to understand system functions

The first point is a complex issue, but mostly stems from the Linux use of package managers. Package management is one of those concepts that seems great on the outset, but fails in practice. The issue is that each package has a complex chain of dependencies unique to itself. In order to be certain that a package is compatible with all installations, all combinations of installed packages must be tested! As it is unlikely that anyone would go through so much trouble, the incompatibilities between packages accumulate, and before long the packaging system is rejecting new installs. And that's assuming that a graphical installer exists!

If a graphical installer does not exist, then life becomes even more difficult for the end user. Instead of launching a GUI and selecting the applications he wants, the user must open a terminal and begin typing cryptic commands for which he has no training for.

Many proponents of packaging systems downplay these issues by stating that packaging errors don't exist on system XYZ (despite proof to the contrary), and that if the user is running Linux he should be "smart enough" to know how to use the command line. Such statements are just silly. Users want the computer to make their lives easier. Any barrier thrown in their way will only drive them to a different platform. Unfortunately, package managers still drive most Linux desktop distributions.

The second point is caused by the spread out arrangement of Linux system files. This arrangement is intended to ease the multi-user aspect of Unix system. Unfortunately, it greatly complicates the user view of the system. Existing users are accustomed to having a root folder with a couple of system folders to worry about. For new users, even one system folder can be a massive issue. (I'm sure we've all heard about or seen the guy who deleted his Windows folder and then expected everything to work properly.) Expecting these users to understand the cryptically named "usr", "bin", and "etc" folders is probably a bit much. (OS X Finder actually hides them.)

Points three and four are caused by some rather interesting decisions in the OSS community, constant arguments about interface design, and beta quality software being bundled in distro releases. (Red Hat is particularly guilty of the latter.) For example, there's a single, non-hidden folder on Windows where Start Menu shortcuts are stored. This folder can be accessed by right clicking on the Start Menu and hitting "Show Folder". While the Windows design is a confused interface, at least it's consistent. In GNOME and KDE, the menu folder has moved several times, switched back and forth between a GUI Tree, File Explorer, and Drag/Drop-on-the-menu interface for modifications, and they had a split personality on whether or not user-specific menu items could be created. Most of these issues have been smoothed out to some degree, but not without leaving users utterly confused.

With these issues in mind, let's put some thought into how we might fix them.

Part 2: Applications

Links:

Making desktop Linux software installation easy
The Year of the Linux Desktop: 2003
2004: The Year of the Linux Desktop?
2004 Won't Be the Year of the Linux Desktop
2005 Will Be the Year of the Linux Desktop
Desktop Linux News, Articles, and Forums

Explanation of Database File Systems

2005-06-09T20:09:00.000-07:00

Category: Technology Explained

One of the hottest topics on the market today is Database File Systems. Between Gnome Storage, WinFS, and now Apple Spotlight, it seems like everyone is making a big deal out of these features. But what is a database file system and why is it so important? I'll attempt to answer those questions and provide a good explanation of how a modern Database File System can be structured.

Defining the Problem

In every day life we tend to keep track of things by association. For example, I know that the bill I received from the credit card company is both a bill and is from my CC company. So what if I wanted to find that bill again, but didn't remember what I did with it? Well, I'd probably go and check in my bills first, then perhaps in a folder where I kept old paperwork, then on my desk, then perhaps somewhere that I might have left it by accident. Note the associations here:

All Bills
Archived Paperwork
Recent Desktop Documents
Search Common Areas

These associations are a form of "meta-data" or "data about data". Usually we don't consider this information anywhere near as important as the data itself, but without it we couldn't even find the data!

Now let's assume for a moment that our mythical credit card bill was delivered to us electronically. If we further assume that we kept it as a file on disk, then the question that comes to bear is: Where do we put it so that it's easy to find?

Options include:

In a "Bills" folder. It's probably a good idea to keep all the bills together. That way we know where to find one that we might need.
In an "Archived Documents" folder. The bill is already taken care of and paid. Why not just archive it and keep it out of the way?
On the Desktop. After all, it's an important file, right? So this important file should go somewhere noticable.

As you may have noticed, this is very similar to the issues faced in real life. The difference is that a computer should be able to do much more to help organize the information than just placing it somewhere you hope is obvious.

To a certain degree a computer can help. If I lose track of what I did with the bill, then I can attempt to search for it. The computer will then trace through every file in the system attempting to find what I lost. The problem with such a search, however, is that the search can only work as well as the name I've given the file. If I got lazy and called the file "CC Bill", then it won't be returned in a search for "Credit Card".

Meta-Data

What if we could improve upon the situation presented above? For example, what if we could place the bill in the Bills folder, Archived Documents folder, and the Desktop simultaneously? How about if we could search inside the document for the information it contains? What if the computer could intelligently score files it finds in a search and sort them based on the files it thinks are the best match? With the meta-data from a database file system, all of this becomes possible.

Generally speaking, there are three types of meta-data that a system might support:

Inherent meta-data
User applied meta-data
Organizational meta-data

Inherent meta-data is data that can be derived from the very existence of the file. At the simplest level, this includes things like the date it was created and the size of the data. More complex meta-data schemes may actually dive into the binary stream of the file data and extract additional meta information assigned to the file. This could be as simple as the text of a word processing document or as complex as the name of a movie or the artist of a song. It should go without saying that the better the system is at understanding files, the more useful information it can extract from a file.

User applied meta-data is data that the computer user explicitly adds to a file. For example, I might type a note in a meta-data field stating that I need to remember that this bill is due by the 28th of February instead of the usual 30th of the month. This information can often be very useful, but it can be difficult to convince users to take the time to apply it.

Organizational meta-data is something of a cross between the two previous types of meta-data. It's the type of data that a user might add to a file for the purposes of better organization, and thus becomes an inherent attribute that can be queried on. For example, let's say that I wanted to categorize bills under a meta-data tag called "Bills". And let's say that I then had a tag for each credit card company so that I could easily find correspondence with them. Now I have added meta-data to the file that tells the system that the document is a Bill from Credit Card Company A!

Out with the Old, In with the New

The key to a database file system is that the meta-data attached to files provides us with a better method for searching for things. The theory is that if searches get good enough, the traditional methods of accessing files can go away all together. Instead of browsing through folders, the user can just type "Bills" and get a list of all the bills stored on the system. Or the user can narrow it down and ask for "Bills from Credit Card Company A" and receive a more specific list of results. The key is that the search will always come back with what the user needs, but in a much quicker fashion than if the user had attempted to manually find the files.

So does that mean that Folders will go away all together? The answer is both yes and no. No, traditional folders won't be as useful. But at the same time, it is occasionally nice to be able to browse the information contained in your system. The replacement solution is two fold. The first part of the solution is to allow the user to save search queries as a psuedo-folder. This provides a user with easy, and automatic organization of his files. For example, he could create a saved query that searches for all movies. Then whenever he wishes to know which movies exist on his system, he can just open the psuedo-folder and see the results of the search!

The only issue with using saved queries is that without regular folders your files will be lost until you can craft a query to find them. As a result, another solution is necessary.

The second part is a concept known as Labels. Labels are a type of meta-data that falls under the Organizational category. The idea is that you can create as many Labels in your system as you'd like, then apply them to individual files. As files are tagged with Labels, they automatically appear in psuedo-folders that display the name of the Label. Files lacking a label will appear in an area that displays unlinked files. This list not only ensures that the user never loses his files, but also encourages him to properly Label them. And since a user can apply and subtract any number of Labels at will, there's much less of a need to ensure organizational correctness up front.

The concept can even be extended into common metaphors in use today. What if the Desktop stopped being a folder for files, and was just a standard system label? Files could easily be moved to the desktop and removed just by applying and removing the label! Files could be trashed with nothing more than a Label called "Trash". Hundreds of uses could spring up to ease the user interface just because a better method of organization exists! That is why database file systems are considered the future of computer file systems.

Links:

Wikipedia: File Systems
Practical File System Design with the Be File System (1999)
Spotlight Technology Brief
DBFS for KDE
Microsoft WinFS

Questions? Comments? Use the "comments" feature on this blog to leave feedback. The more people who I know are listening, the more of these articles I will do. So sound off and let your voice be heard!

Have you been waiting for Star Wreck?

2005-06-02T20:43:00.000-07:00

Category: Entertainment

After 5 years(!) of development, the legacy of Star Wreck 6 is finally coming to a head. No, they have not given up on the project. Something far, far better has happened. According to the front page, Star Wreck 6 will hit DVD distribution on August 20, 2005! So get your twinkler beams ready folks, because we're in for a Star Trek parody of far greater proportions than any Star Trek movie ever released. Move over Kirk, you're in Captain Pirk's seat!

For those of you unfamiliar with Star Wreck, here's a quick history: Back in 1992 a young Finnish boy by the name of Samuli Torssonen decided to do a video of a starship battle. Utilizing a copy of Deluxe Paint for frame by frame animation, he managed to put together a four minute clip of various Star Trek ships doing battle. Just to have fun, he also added a comedic aspect to the film. Boy, if this young fellow had any idea what he was getting into...

The success of the first video spurred young Samuli, and by 1994 he'd managed to pull off a new Star Wreck video. But this time Mr. Torssonen did the ships in 3D, and used cell animations for the characters. The result was one of the funniest Star Trek parodies ever to hit video!

The laughes and running time only increased in Star Wrecks III and IV, jumping all the way to 31 and 47 minutes, respectively. Star Wreck was becoming a feature film, and Samuli only wanted to push it farther. Thus Star Wreck V: Lost Contact made a huge leap and became the first live action Star Wreck episode. Only one problem....

Our heros were trapped in the past....

That was 8 years ago. Plans for a sequel had been immediately put into action, but the project was progressing slowly. The decision was made to make the next episode into a feature film, but the team was unsure if they could pull it off. As a result, Samuli created a secret project known as Star Wreck 4 1/2. This project tested new camera techniques and 3D rendering packages to prove that they could make a professional quality movie. The result was short and not as funny as the previous installments, but was an amazing sight to behold. The quality of the 3D was as good (if not better!) than the quality of The Next Generation TV show! Not to mention the compositing done for the bridge scenes! For the first time, the live action bridge was truely open, and appeared to be much more than just an image pasted behind Captain James B. Pirk.

The triumph of the 4 1/2 project was both a blessing and a curse for the sixth Star Wreck installment. Most of the film that was shot to date was scrapped and reshot. New actors and actresses were brought on board, the music and 3D were redone, and the greatest comedy adventure of all time was on its way to becoming a reality.

So load the light balls, and get ready for an action packed adventure! Coming August 20, 2005 to a DVD player near you is:

Star Wreck 6: In the Pirkinning

Fukkoooooooovvvvvvv!!!

Links:

Star Wreck 6
Movie Trailer
Previous Star Wreck Movies
Star Wreck: Asskicker (A spin off)

Another Look at Java Desktop System Release 2

2004-06-04T14:06:00.000-07:00

Category: Operating Systems

Note: I originally intended to submit this to Linux.com or some other online news agency. Since I've recently become quite busy with more lucrative writing, I've finally decided to release this since I'll never finish polishing it. Enjoy, and feel free to share your thoughts.

Linux.com's recent review of Sun's Java Desktop System came up with less than stellar results. When my colleague-in-print attempted to take the OS through its paces, he found that he was unable to even complete a simple installation. In his review, he identified the culprit as Sun's poor support for ultra-modern hardware. (Although many Slashdotters have commented that it may have been due to bad CDROM media.) Given that Sun has often claimed that JDS is targeted at corporations users who wish to reuse existing hardware, I will attempt to provide a second look based on slightly older hardware.

Configuration

For this test I am using a PIII 733 w/512 MB of RAM, a GeForce2 GTS, an AWE32 sound card, an Intel Etherexpress NIC, two 40 GB Western Digital ATA100 drives, a Creative 40x/8x/4x CDRW, a standard PS/2 keyboard, and a Microsoft Intellieye Optical Mouse. This machine was custom built a few years ago for maximum capacity and compatibility, with a focus on machine stability rather than performance. These decisions resulted in a slower processor and smaller disks than the maximum available at the time. In other words, exactly the type of machine a corporate office might look at deploying.

Installation

Upon inserting the disk and rebooting, I was presented with a graphical GRUB bootloader screen. I selected "install" and the YaST installer started up. If you've ever used SuSE, you should know what to expect out of YaST. It does a quick analysis of your system and suggests intelligent defaults for the install. When I previously installed JDS Release 1, the installer automatically chose ReiserFS as the filesystem and detected all the hardware without issue. The same thing occurred this time, except that I told the installer to upgrade the current installation instead of reformatting my ReiserFS partition.

After the installer finished analyzing the system, it proceeded to install all the packages from the first disk. This appeared to be core system components such as the kernel and command line utilities. The installer then asked me to remove the CDROM and reboot. After the reboot, the installer relaunched and began installing packages from the second and third CDROMs. When it was finished, I was presented with the Java Desktop login screen.

All in all, it doesn't appear that the installer has changed from Release 1 to Release 2. The install was straightforward, easy, and intelligent. I can't say that I have any complaints in this area, save for the fact that the install "feels" very much ripped from SuSE rather than having been customized for JDS.

Overview

Upon login, you are presented with a highly customized GNOME interface that looks a lot like Windows. There's the "Launch" button on the lower-left hand corner, volume control on the right, and the list of open windows in the middle. There are only two things about the interface that would seem unfamiliar to a Windows user. The first is that the clock is located directly to the right of the Launch button. This is pretty minor and will only confuse users for a short while. The second difference is the addition of a desktop pager. This is potentially dangerous feature as help desks are bound to receive a large number of calls from users who believe that it caused all their programs to crash. Thankfully, it can be easily removed via the right-click popup menu.

A useful assortment of both Open Source and commercial software can be found under the "Launch" menu. All the standard GNOME utilities and games are there, as well as Mozilla, Real Player, Macromedia Flash, StarOffice, Evolution, GIMP, Java Media Player, and a variety of open source Java software. Given the visual integration of the Desktop and GNOME software, Mozilla and RealPlayer seem out of place. Mozilla has a skin to make it look like a JDS application, but the skin is only half complete. In all fairness, it has been much improved since Release 1. Never the less, there are still a lot of visual glitches and combined button styles.

RealPlayer is the same free player that is available for download from the Real site. As one would expect, there is no full screen support, and it does not integrate with the desktop in any way other than to associate itself with RealPlayer files. Sadly, not even the file associations works correctly. Double clicking on a RealMedia file merely results in an error message about expecting a file path instead of a URL. RealPlayer then forces you to navigate its antiquated File|Open menu in order to open the file.

The Java Media Player is pretty much useless. Of all the videos on my NTFS drive (including professional videos of real events, recordings of television shows, and VCD cartoons for the kids), only some MPEGs (excluding the VCDs) and a few documentary clips of shuttle launches were able to play. MP3s do play, but JMP doesn't have a playlist to queue them up. On top of that, JMP automatically loops both audio and video files. In short, Sun needs to spend a little time working out a few of the kinks in the player.

StarOffice is nearly identical to the latest OpenOffice release, sans two minor differences. For one, the look and feel has been smoothed out to give it a more professional feel. Second, the clipart library has been upgraded with a wide variety of images for spicing up any document or presentation.

As it turns out, Evolution is a very slick email client. The email setup is reminiscent of Outlook Express, but with the addition of a zooming world map for choosing your physical location. I have no idea why Evolution needs geographical information, but the map widget is so impressive that you'll find yourself not caring. Once Evolution is configured, the interface looks and functions almost exactly like that of Microsoft Outlook. Weather, news, calendar, mail, and contacts lists are all there and function as one would expect. A few features for grouping mail appear to be missing, but this is far from a deal breaker. Even advanced users of Microsoft Outlook should feel right at home.

GIMP is included as an alternative to Photoshop and MS Paint. While the version number is 1.3, the interface is the same as the new 2.0 series of GIMP. Sun must have decided to include a well tested development branch that lead to the 2.0 release.

Java Desktop System also adds a desktop folder called "Network Places" that's very similar to the Windows' Network Neighborhood. It allows you to access various network file systems including FTP, SMB, and NFS. Mounting one of these file systems simply adds a shortcut to the "Network Places" folder instead of actually mounting the it to the underlying filesystem. This has the side effect of preventing most programs from opening and saving files to these locations. Attempts to open a file are usually met with the message "[program] does not support [ftp|smb] urls". To get around this issue, I found myself copying files to and from the desktop to edit them.

SMB (Windows File Sharing) was particularly annoying. The Network Browser doesn't work without a Domain Controller, and every time I changed directories or copied files, I found myself facing three or four "enter password" dialogs. While I was able to muddle through, businesses may wish to consider adding smbfs entries to the '/etc/fstab' file instead.

Java Support

Java support in Release 2 is overall a very pleasant experience. Applets work without any configuration, and Java Web Start programs launch at the click of a link. Even executable JAR files work correctly, which is a major improvement over the previous version of JDS. Given that JDS has no native installer or packaging system, the support for executable JAR files will finally allow ISVs to deploy software on the Java Desktop System.

New in this Release

Java Desktop System Release 2 comes with a mixed bag of new features and broken software. On the up side, JDS now allows you to create and modify Launch Menu icons by right clicking on the open menu. The new Online Update program is very cool looking, and easy to use. It also adds a CD with the Java Development Kit (JDK 1.4) and the NetBeans IDE. Unfortunately, it does a lot of things quite badly as well.

The first thing I noticed was that the graphical boot has disappeared. Where I used to see an att attractive progress bar while booting, I now see a small icon of Tux that quickly scrolls off the screen as the system boots. I'm sure that Sun had their reasons for this, but it makes the system look much less professional.

The next thing I noticed was that several programs were broken. The Online Update program appears to replace the "Online Software Update" program from the previous version. Too bad Sun mislinked it to consolehelper instead of consolehelper-gtk. Perhaps this was an artifact of doing an update instead of a clean install, but somehow I doubt it. It would be far more likely that the link simply wouldn't exist. It's a good thing that Sun is deploying this to corporate users. Home users would have no idea how to fix such a problem.

Sun also added two icons for "Sun Instant Messenger". The first one is a link to a JNLP file that supposedly launches the software. This returns a 404 Not Found from Sun's website. The other icon is a link to sign up for the service. If you follow the link and sign up for the service, you are presented with two launch options. Supposedly one is over a secure VPN. The "non-VPN" link points to an unsigned application. I don't know what they're thinking, but Java Webstart won't launch an unsecured application that asks for unrestricted access to the machine. Launching the link for the VPN version fails with a missing library. Do they want people to try this service or not?

The last change is pretty minor. Totem (listed as "Video Player") seems to have completely disappeared from the system. While the icon for it remains, GNOME cannot find the executable. Again, this may be from the fact that I did an update instead of a clean install. It really doesn't matter as Sun's release of Totem has very few advantages over the Java Media Player.

Final Thoughts

Despite the rather "rushed" feel to Release 2, the Sun Java Desktop System is slowly shaping up into a serious Corporate competitor to Microsoft Windows. Many managers will probably decide that they wish to stick with their Windows laptops for the time being, but they'll probably drool at the opportunity to replace the rest of their very expensive Windows infrastructure. Home users might also appreciate this OS in its prebundled OEM form since its similarity to the Windows interface makes it easy to learn and use. However, they may wish to wait a few releases for Sun to work out the bugs.

Java Desktop System: A Day Late, but On the Money

2004-05-17T17:02:00.000-07:00

Category: Operating Systems

Sun has long been criticized for its failure to jump on the Linux bandwagon. Open Source advocates often point to Sun's competitors as an example of why Sun should embrace Linux. With the release of the Java Desktop System, customers can now rejoice at Sun's willingness to please. But does this desktop stack up to the competition, or will it fail as surely Corel Linux did? To answer this question, I purchased Sun's Java Desktop System and took it through its paces.

Installation

The OS came on a total of 6 CD-ROMs, all packed into a standard Sun media kit folder. As is customary with modern distros, the first three CDs contain the binary packages, while the fourth and fifth CDs contain the source code. The source CDs are particularly necessary if you wish to install third party kernel modules. (e.g. The official Nvidia driver.) The sixth CD is merely a collection of various open source and Sun documentation packages. I personally didn't find the documentation CD to be of much value. Of particular annoyance was the fact that all the screen shots on the were for a vanilla GNOME desktop or SuSE install. None of the documentation was really JDS specific.

Installing the Java Desktop System was quick and easy. All of my hardware was detected automatically (including my ISA Awe32 sound card), and JDS chose intelligent defaults for everything. That includes choosing the ReiserFS format by default (a very sensible choice). Beyond that, but there really isn't much more to say. After it autodetected everything, I clicked Next and it began installing the OS. A reboot later I was sitting at the fully installed desktop. Overall, a very pleasant experience.

Java Integration

Despite the name of the product, Java integration is something of a mixed bag. On one hand, browser integration works great. Applets work without any special configuration, and Java Web Start applications can be launched and/or installed with a single click of the mouse. On the other hand, support for standard Java applications is very poor.

The first problem is that only the JRE is included. This makes it impossible to do any Java development without downloading and running a full JDK install. Even worse, executable JAR files open in the archive manager instead of being launched with the 'java -jar' command. This is inexcusable for an operating system that carries the Java name. The final annoyance is in the default Swing look and feel. Sun has a GTK+ look and feel, but the default OS L&F is the standard Metal L&F. The result is that Java applications stick out like a badly bruised thumb.

Thankfully, there is a silver lining. Sun has promised to add developer tools to the May release of JDS (not available as of this writing). This package is supposed to include the JDK, Java Studio, and NetBeans 3.6. It will be available to anyone who payed for a previous version of JDS. With any luck, that version will transform the OS into a much more attractive option for both developers and ISVs.

Software

The Java Desktop System comes with a useful assortment of both Open Source and commercial software. All the standard GNOME utilities and games are there, as well as Mozilla, Real Player, Macromedia Flash, StarOffice, Java Media Player, and a variety of open source Java software. Given the visual integration of the Desktop and GNOME software, Mozilla and RealPlayer seem out of place. Mozilla has a skin to make it look like a JDS application, but the skin is only half complete.

RealPlayer is the same free player available for download from the Real site. As one would expect, there is no full screen support, and it does not integrate with the desktop in any way other than to associate itself with RealPlayer files. Sadly, not even the file associations work. Double clicking on the file causes Nautilus to pass RealPlayer a URL instead of a file path. Predictably, RealPlayer chokes on this URL and forces you to navigate its antiquated File|Open menu.

Third Party Software installation is even more problematic. Sun's version of YaST 2 is sufficient to install only a small subset of Open Source software available for Linux, as well as automatically install the latest security updates. In theory, new versions of the OS will also be delivered via the same interface. Beyond that, there are no graphical tools for software installation. Even SuSE's YaST interface for installing RPMs appears to have been removed. This leaves the user with the choice of either running 'rpm install' manually, or compiling software from it source.

When I spoke with Sun on the issue, they acknowledged that it was problem and suggested that I try a Java product such as Zero G. Unfortunately, Sun's failure to add support for executable JAR files means that even products like Zero G won't be able to create installers that launch from the GUI. Hopefully this oversight will be corrected in the near future.

On the bright side, the Nautilus file manager has been configured to be very similar to Windows Explorer, and should feel perfectly natural to any Windows user. Only the highly attractive icons and in-icon preview mode keep you from forgetting that you're not in Windows.

The GNOME interface has also been tweaked to feel very much like Windows. The standard foot menu applet has been restyled into a Launch button that's reminiscent of the Window's Start menu. To the right of the Launch button, you have your system clock, the list of running programs, then the system tray icons (including the Volume Control). Other than the clock being in an odd place, this should all seem very natural to a Windows user. The desktop pager might cause some confusion, so companies may prefer to disable it before they receive the inevitable my applications disappeared! tech support call.

Multimedia

In the past, many desktops have used their business focus as an excuse to include reduced multimedia support. Normally I'd simply make a note of it and move on. In JDS's case, Sun has also made an effort to target home users through OEM machines sold at WalMart. It's possible that Sun has added extra multimedia capabilities to the OEM install of JDS, but I do not have such a machine to verify this supposition. Thus I must to proceed on the assumption that the OEM version of JDS is no different than the Media Kit.. This is rather unfortunate for Sun as multimedia support is one of the weakest areas of the Java Desktop System.

JDS includes the Java Media Player, GPlayer, and Totem. Double clicking on an MP3 brings up the Java Media Player. The upside is that JMP plays MP3 files. The downside is that each instance is only good for playing one file. Play lists are out of the question, and the default is to continuously loop on the same file.

None of the Media Players I tried would play any of the AVI files I had available to test with. Since most AVIs today are encoded in patented formats like MPEG4, this came as no surprise to me. Most Linux distros seem to be attempting to avoid patent issues in a similar fashion. Given how large of a company Sun is, it would be a nice surprise for them to simply license the patents required to legally ship non-crippled software.

MPEG2 files were associated with Gplayer. These seemed to play reasonably well. I had very few MPEG2 files on hand, so I found that other than testing there wasn't much use for this support. I do not have a DVD drive, but it's possible that this program is for current or planned DVD support.

3D support is non-existent. In fact, Sun has completely removed the GLX extension. I only realized this omission after downloading the NVidia Linux drivers from the official site (only the generic 'nv' driver is included in JDS), and attempting to run a few JOGL (Java OpenGL) programs. I was a bit surprised when the first program immediately exited, and the second program hung. A quick check of the console confirmed that the GLX extension couldn't be found.

In the end, I performed a command line install of XMMS and Xine. This was sufficient to make use of my multimedia collection. 3D games are still out of the question.

Overall Impression

The interesting part about the Java Desktop System is that despite all it's flaws, it's actually a very pleasant and usable system. I would even go as far as to say that it reminds me a lot of when I first started using Windows 95. Everything is new and shiny, the desktop is pleasant on the eyes, the GUI is responsive, and you don't feel the overwhelming need to drop to a terminal. Given that the JDS interface is supposed to be ported to Solaris, it may be that Sun intends JDS Linux to be to Solaris as Windows 9x was to NT.

The downside of such a comparison is that JDS Linux is just about as quirky as Windows 9x. Putting an Audio CD in the drive once caused the CD Player to hang and the rest of the system to respond in short spurts. Opening and closing the CD Drive resolved the problem. Another time I was doing an 'rpm --rebuild' in a terminal (JDS uses GLIBC 2.2.5 instead of the latest 2.3) and the GUI became sluggish and unresponsive. Data CDs will frequently fail to eject, supposedly in use by some mysterious program. None of these quirks are insurmountable, but they do give the OS a beta quality feel to it.

JDS does have a few things in its favor that Windows 95 didn't. Unlike Microsoft, Sun releases updates to their software on a quarterly basis. This means that major OS fixes are (hopefully) no more than 3-6 months away. JDS also has the advantage of being able to pull from a large base of existing software. Whereas Microsoft had to build everything for themselves (admittedly a doable task for such a large company), Sun can pick up an Open Source project that meets their needs, add some polish, submit the changes back to the project, and release it with the next update of JDS.

Wish List

Besides the standard list of OS improvements (e.g. Make it faster, support more hardware, bundle more software, etc.), I believe Sun could really improve their OS by making use of some of their business connections and clout as a large corporation. For example, Sun should license the MPEG4 and/or DivX codecs. That alone would make Sun the only Linux company to distribute a working media player. Next on the list, they could convince Adobe to create a version of Acrobat that would integrate with the GNOME desktop. Even a RealPlayer version that supported full screen would be a welcome change.

Beyond that, I think that Sun really needs to take the lead on software installation. Packaging systems are great for keeping system libraries and command line programs in sync. Unfortunately, installing a GUI program ends up being a frustrating experience fraught with dependency hell. Simply defining a new standard for GUI software installation (preferably one that included all dependencies) would go a long way toward making the system more usable to the technology challenged. Even those of us who are experienced with Linux would readily appreciate being saved the trouble of tracking down dependency after dependency.

Final Thoughts

If you're a business looking to break your Microsoft habit, or a Linux user who's looking for a distro with a comfortable GUI, then JDS is definitely for you. If you're a home user looking for an OS that's cheaper and easier than Windows, then I suggest that you wait a few releases before making the big leap.

One way or another, JDS represents a tremendous shift in usable Linux Desktops. I applaud Sun's efforts and eagerly await their future offering.

To SuSE or not to SuSE? Is there even a question?

2004-04-12T14:22:00.000-07:00

Category: Operating System

If you haven't read my first entry, I would be much obliged if you did. It gives the setup for this series of articles. While you're at it, you might even go back and read the Mandrake review as well as the comments.

So here I am. A SuSE 9.0 boot CD in my hand, and a desire to find a Linux distro that works. I'll be damned if SuSE isn't off to a great start. They *only* allow Internet installs via a small boot ISO. And unlike the Fedora boot.iso, this thing has everything. Installation, Rescue CD, boot manager, etc. I might just have to keep this thing around for the next time an OS goes haywire.

The installation proceeded smoothly, right after I got over the initial confusion caused when the CD asked for the IP Address of the FTP server. It turns out that you can enter a hostname just fine. (A hint: Make sure you write down the name of your mirror server, as well as the directory SuSE is in before you start the installation. There's no predefined list of mirrors to choose from.)

The installation GUI was slick, with it autodetecting just about everything. I had to change a few setting (such as installing GRUB on

/dev/hdb instead of /dev/hda), but I can't say that I've ever seen an easier install process. It even chose ReiserFS as the default. If I had one complaint, it's that every operation takes it forever. But it's an install. You're supposed to do it once and be done with it.

Once installed, I rebooted only to see "GRUB" on the screen and nothing else. Thankfully this great install disk of SuSE's allows me to boot into an existing OS directly from the CD. Once I was in, I played with GRUB until I got it to work. It turns out that the BIOS informs GRUB that the drives are backwards when booting from the second drive. Thus (hd1,5) (where it was looking) was actually (hd0,5), and Windows was (hd1,0). I changed the menu.lst file and everything worked fine.

I suppose I should probably be annoyed by this problem, but I'm not. Given that I'm installing these OSes onto the second partition of a secondary slave drive, I'm willing to cut them some slack.

When the system came up, everything pretty much worked. All my NTFS drives were mounted, the sound worked, I could play MP3s, my TV card worked flawlessly, etc. Things that didn't work were the mouse wheel, and the video player. The mouse wheel was easily fixed by adding in the "ZAxisMapping" setting to the XF86Config file. The video player played sound, but there was no picture. This really didn't bother me too much as I was planning to install VLC.

Which brings me to my next point. Why does every Linux install have to involve RPM hell? I had to pull 15-20 RPMs from various sources in order to get VLC installed. Most home users won't know what "libtheora" is, where to get it, or how to install it. Not to mention scary names like "libart_gpl.0". Linux systems work fine for the software that's in their catalog, but anything even slightly "different" becomes a real pain in the ass for users. It's time to decide whether Linux wants to be a "hacker's system" where everything is compiled from source, or a "desktop system" where binary compatibility and simple installation is a must. So far, Linux has been targeting the "Workstation" market which just isn't good enough. FreeBSD, Windows, and Mac OS X will eat Linux for lunch in that market.

Anyway, after mucking around with half a billion RPMs, VLC installed and even put itself into the SuSE start menu. (A nice touch.) I then loaded one of the kids' cartoons off of the NTFS drive, and it played flawlessly.

I still have to set up the NVidia drivers, but otherwise the system is working beautifully. My only complaint is (wait for it) my mouse still locks up!!! Ok, it doesn't happen as much on SuSE. In fact, I thought that the problem didn't exist until I was mucking around in the hardware GUI trying to make the GUI install my mouse wheel. That was the first time it locked up, and I figured that as long as I didn't mess with any hardware, I'd be okay. Well, it eventually locks up anyway. This seems to be an epidemic with the Linux kernel.

Please! If anyone knows how to fix the mouse problem, tell me! This is the only major issue I'm having with SuSE!

Final Verdict: I think we have a winner. There's still a few areas that would be difficult for the average user (e.g. Setting up the mouse wheel, installing the NVidia drivers, and dealing with the installation of non-catalog software like VLC), but overall it was easy to set up, and the YaST2 software library made software installation a breeze. I'm very pleased with SuSE, and would recommend it to anyone looking to find an easy to use Linux system.

The Mandrake 10.0 Experience (TM)

2004-04-11T14:02:00.000-07:00

Category: Operating Systems

Perhaps I was simply expecting too much out of a RedHat derivative, but my review of Mandrake did not go well at all. I've long heard how easy to use Mandrake is, and how easy it is to set up. Supposedly, you install it and boot. It's that easy. I just wish it were that easy for me.

I downloaded all three ISO images and burned them to CDRW. (I'm just glad I didn't waste CDRs on this.) I then rebooted and began the installation. A very nice looking GUI installer came up and I began to follow the steps. Unfortunately, the installer locked up on the first button I clicked. I had to reboot a few times and keep trying before I finally figured out that this is the same mouse bug as in RedHat and Fedora. I was then able to get through the installer by careful use of the keyboard and mouse. (Hint: Click the button and let go of the mouse until the operation is completed. If you move the mouse, you're dead.)

The upside of the install was that it was very easy to choose the packages I wanted, and the new 2.6 kernel was installed without fuss. The installer even warned me that it would have to downgrade to 2.4 if I wanted to install the Linux Standards Base (LSB) software. This was definitely a nice touch, as I wanted to use the system as Mandrake had intended. (I was also hoping the 2.6 kernel would get rid of that $%Q# mouse problem.) I was also able to easily select ReiserFS as my default filesystem.

The downsides to the install was that there was no small boot CD to install over the internet (all 3 CDs were required), it wrote the Linux partition as a non-bootable extended partition (which the Mandrake version of the bootloader somehow got around), and the sound card couldn't be configured (it told me to run 'sndconfig' after the first boot). Considering that sound was exactly the reason I was performing this experiment, this did not make me happy.

After install, the machine booted up to KDE 4.2. I'd love to give you a guided tour of all the great new features, but to be perfectly honest I never got to find out what they are. The same mouse locking problem that was present in Fedora was also present here. That was annoying in of itself. Then I spent time trying to find this 'sndconfig' utility that I was supposed to run.

I eventually found it under '/usr/sbin' and ran it. It was unable to detect my PnP sound card, and asked me to manually enter the values. I did so, and the sound card tested fine. Upon exit, the utility froze up and I was forced to hit CTRL+C. The '/etc/modules.conf' file appeared to have been properly modified, so I attempted playing a sound. Nothing but an error about sound not being configured. So I restarted the system. When it came back up, I still didn't have sound. I reran the 'sndconfig' utility with the exact same results. I then checked and verified that the audio streams existed under the '/dev' directory. There were there. I then played with the KDE sound config to get it to recognize the sound device. Even manually setting the path to the device didn't work. So, I restarted the system again, hoping this time it would work.

I then made a startling discovery. After I reboot the machine, the sound devices disappear from the '/dev' directory! They only come back after I rerun the 'sndconfig' utility. I then figured that I had better check Mandrake's website to see if they had any help on the problem. Unfortunately, I found I was unable to reach their website. In fact, I then realized that I couldn't reach any website. It seems that after one of the reboots, the network card somehow disabled itself. That's about where I gave up.

Final verdict: I don't have time for this.

I sincerely hope that the rest of you have a much better experience with Mandrake 10.0 than I did. I really wanted Mandrake to work, but it simply ended up being too painful to continue.

Finding the best Linux Distro

2004-04-08T12:15:00.000-07:00

Category: Operating Systems

Well, Windows 2000 has done it again. My volume control disappeared a while back, so I uninstalled and reinstalled the drivers. Now the audio drivers are so screwed up that I get NO sound, despite the drivers being installed just fine. The problem appears to relate to the Preferred Devices settings, which are now grayed out.

Since my kids really like watching cartoons on the computer, I immediately started looking for an alternative way to play them. My first attempt was to use the Java Desktop System demo and Knoppix CDs that I have. Both have media players, but neither one can handle the encoding format. Since I know that VideoLAN can handle it (that's what I use on Windows), I tried to install that. Unfortunately, you can't take one of those CDS through an entire build cycle. There's simply too many packages to install to be doing it every reboot. So, I decided to try out different Linux distributions.

I've decided that I want a Linux distro that works correctly out of the box, and needs minimal configuration. If I have to start recompiling stuff, or run into major technical hurdles, it's gone. In the past, technical issues have always driven me back to FreeBSD. It may take slightly longer, but at least everything works when my BSD system is set up. But for this experiment, I'm valuing expediency and ease of use above all else. Supposedly, Linux can provide this.

My configuration:

PIII 733 w/Asus MB
GeForce 2 GTS
2 identical 40 gig drives (~12 GB is set aside on the second drive for alternate OSes)
Intel Etherexpress 100 NIC
ISA Awe32 Sound Card
Pinnacle PCTV Tuner
Microsoft USB Optical Mouse w/Wheel (Intellimouse)
PS/2 Keyboard
15 inch Dell Monitor
Creative 8x CDRW drive
No floppy drive
512 MB of RAM

This machine is over 3 years old at this point, and has been holding up pretty well. I custom built it for capacity and compatibility (except for the AWE32 that I yanked from an old computer), so it's probably got another 2 or 3 years of life left in it.

On to my first victim! Err... distro!

Fedora Core 2

In the past, my impression of RedHat has always been that it is a very pretty OS on the surface, but with a lot of hidden terrors under the hood. I won't go into my long list of complaints concerning RedHat as it really doesn't make a difference here. I was hoping that Fedora Core would meet my requirements, and I won't have to worry about it's idiosyncrasies.

The install was very easy and smooth. The GUI was responsive, and gave me no trouble what so ever. I told it where to install, and it did it. I used the boot.iso disk to install over the internet, but it didn't seem to impact anything over using all 3(!) CDs.

Once the install was complete, the system came up and allowed me to log in. I typed in my username and password, excitedly started trying to use it, and then...

my mouse locked up. *sigh*

Last time I tried RedHat 8, the same thing happened. You could use the system for awhile, and then the mouse would simply stop responding. If you tried not to move it during IO operations, it could be made to live on for quite awhile. Unfortunately, it always locks up in the end. Then I have to use the keyboard to reboot and pick up where I left off. The fact that this problem is *still* happening after 4 versions, is not encouraging.

So, I rebooted and started over. Next task was to access the videos on my NTFS drive. Oddly enough, the drives weren't already mounted. How odd. So I checked around the GUI, and found that the drive didn't show up in the configuration tools. Alright, down to the terminal shell. I type 'mount -t ntfs

/dev/hdb1 /mnt/multimedia' and I get 'type NTFS is unknown'. What in the world?

A quick check of the net shows that NTFS is not supported by Fedora Core 2. Something about being "freer than free", or some other diatribe. The Linux NTFS project says that they do not support Fedora Core either. Add to that the fact that NVidia drivers, MP3 playing, and other core niceties are missing (oops, mouse locked up again, reboot), this one's dead on arrival.

On to Mandrake!