Another dusty benchmark: Filesystem Torque Curves, gentoo 2.4.18

Filesystem Torque Curves, gentoo linux 2.4.18 5400rpm maxtor 80gig on athlon xp1700

Doing more cleaning, and dusted off an old filesytem benchmark deserving of notice.

This benchmark was built from a bash script that formatted the same partition each time with different filesystems and the -o options shown on the legend.

dbench was run with x processes and then the filesystem was wiped and formatted as above and the buffers purged. I ran this many times and got comfortable with the relative numbers being consistent, so this may represent only 1 test run but it was pretty stable at the time.

dbench runs a recorded session of deliberate samba abuse * n processes; A collection of large file operation and inode abuse.

Presently this machine is an anonymous motherboard sitting on a shelf in a stack next to a collection of anonymous old hard drives these days.

I wrote this benchmark to see if I was actually missing out on something that XFS brought to the table. I tend to go with reiser3 (tails rock for all things portage) for general purpose filesystems and ext2 or huge swapfs for performance workloads such as processing multimedia or mythtv sandbox.

XFS was touted as the coolest filesystem and supposedly excelled at all things for all needs according to several zealots I've run across. I always thought that it looked a bit contrived compared to ext2 and had less thought to detail than reiserfs. i had no idea those 2 filesystems would leave it in the dust until I ran the benchmark.

I'm fascinated these days with LinLogFS -- A Log-Structured Filesystem For Linux or Linux Log-structured Filesystem Project but the applications haven't presented themselves frequently.

Other items of note from this benchmark

• dbench probably offers some opportune areas for inflated numbers corresponding to the force of numbers of developer man-years exposed to the linux driver module in question... where..
  
• ext2's implementation is brutally terse with good results and lots of eyes and hands contributing vfs enhancements based on ext2 being the gold standard to cater around. (vfs doesn't seem so simple since this benchmark's kernel version)
  
• Reiser shows the oddest process load harmonic. there's some nitrous in concurrent power of 2 access there...
  
• JFS was having a really awful hair day at the time.
  
• Minix crapped out on more than 2 processes effectively breaking the script and 0's were padded in.
  
• pagesize as blocksize makes all the difference on fsbench.
• journaling overhead seems to be an option that's somewhat costly for general diversified workloads. if you dont care about the data persistence, then you probably want the fast edition of the data, and won't need a fsck when a mkfs is plenty sufficient and takes a fraction of the time (such as with high performance computing processing nodes and various web state).

PDF: filesystem benchresults
Google Doc: spreadsheet include interval timings

System.Nanotimer benchmarks

i wrote a small java benchmark to see grab the System.nanoTimer value. On this machine there's like a hard 558 ns limit on the x64 jdk 1.6._10 and 698 lower limit on the 32 bit jre.
In this test I also tested out the atomics and the difference between capture times of buffer and arrays. buffers and arrays are virtually identical performance surrounding the nanotime pull

Directbuffer storage access seems to be the consistent lagger, as is expected. The atomics in the java.utils.concurrent.atomic.LongArray class is not so fast ... Boxing also has a lower average than norm.

of note: the jit works well, they all tend to share the lowest access times(hard limit mentioned above)

it looks like factoring out the call to System.nanoTimer will shave perhaps 500 or so nanoseconds from each loop iteration.

edit:
it was noted from others that variance was pretty high here... the goal of the effort was partially exploring JIT local mediums as well.

Further bench results will probably have some cool visualizations to check out.

see here for source:
http://www.google.com/notebook/public/18239550943564485025/BDQ5wIgoQxbPnoIIj#SDR5nIgoQlITsoIIj

O:\opt\java\jdk1.6.0_10\bin\java -Didea.launcher.port=7536 -Didea.launcher.bin.path=M:\opt\JetBrains\intellij-702\bin -Dfile.encoding=windows-1252 -classpath O:\opt\java\jdk1.6.0_10\jre\lib\charsets.jar;O:\opt\java\jdk1.6.0_10\jre\lib\jce.jar;O:\opt\java\jdk1.6.0_10\jre\lib\jsse.jar;O:\opt\java\jdk1.6.0_10\jre\lib\management-agent.jar;O:\opt\java\jdk1.6.0_10\jre\lib\resources.jar;O:\opt\java\jdk1.6.0_10\jre\lib\rt.jar;O:\opt\java\jdk1.6.0_10\jre\lib\ext\dnsns.jar;O:\opt\java\jdk1.6.0_10\jre\lib\ext\localedata.jar;O:\opt\java\jdk1.6.0_10\jre\lib\ext\sunjce_provider.jar;M:\work\enigmatrie\target\production\enigmatrie;M:\opt\JetBrains\intellij-702\lib\junit.jar;M:\opt\JetBrains\intellij-702\lib\idea_rt.jar com.intellij.rt.execution.application.AppMain com.glamdringinc.benchmark.SystemNanotimerCapture

[{{tester; stat_arr} {avg; 663} {low; 558} {mid; 629} {hi; 13969} {variance;6705}
, {{tester; stat_fin_arr} {avg; 664} {low; 558} {mid; 629} {hi; 22419} {variance;10930}
, {{tester; arr} {avg; 667} {low; 558} {mid; 629} {hi; 31499} {variance;15470}
, {{tester; vol_arr} {avg; 676} {low; 558} {mid; 629} {hi; 38482} {variance;18962}
, {{tester; fin_arr} {avg; 677} {low; 558} {mid; 629} {hi; 48610} {variance;24026}
, {{tester; heap_buff_absolute} {avg; 683} {low; 558} {mid; 629} {hi; 33314} {variance;16378}
, {{tester; array_backed_buf_absolute} {avg; 689} {low; 558} {mid; 629} {hi; 33594} {variance;16518}
, {{tester; atom_arr_lazyset} {avg; 697} {low; 558} {mid; 629} {hi; 39739} {variance;19590}
, {{tester; array_backed_buffer} {avg; 698} {low; 558} {mid; 698} {hi; 38203} {variance;18822}
, {{tester; vol_val} {avg; 710} {low; 558} {mid; 698} {hi; 42952} {variance;21197}
, {{tester; atomic_arr_cmp_set} {avg; 714} {low; 558} {mid; 698} {hi; 39739} {variance;19590}
, {{tester; atomic_arr_set} {avg; 719} {low; 558} {mid; 698} {hi; 87022} {variance;43232}
, {{tester; heap_buf_unbox} {avg; 723} {low; 558} {mid; 698} {hi; 65302} {variance;32372}
, {{tester; mmap_buf_get_prv} {avg; 725} {low; 558} {mid; 698} {hi; 65790} {variance;32616}
, {{tester; atomic_arr_getset} {avg; 728} {low; 558} {mid; 698} {hi; 35479} {variance;17460}
, {{tester; dir_buf_get} {avg; 742} {low; 558} {mid; 698} {hi; 41765} {variance;20603}
, {{tester; mmap_buf_get_rw} {avg; 751} {low; 558} {mid; 698} {hi; 65721} {variance;32581}
]

Process finished with exit code 0

What happened here?

Origins

Main article: SN 1054

First observed in 1731 by John Bevis, the nebula was independently rediscovered in 1758 by Charles Messier as he was observing a bright comet. Messier catalogued it as the first entry in his catalogue of comet-like objects. The Earl of Rosse observed the nebula at Birr Castle in the 1840s, and referred to the object as the Crab Nebula because a drawing he made of it looked like a crab.^[4]

In the early 20th century, the analysis of early photographs of the nebula taken several years apart revealed that it was expanding. Tracing the expansion back revealed that the nebula must have formed about 900 years ago. Historical records revealed that a new star bright enough to be seen in the daytime had been recorded in the same part of the sky by Chinese and Arab astronomers in 1054^[5][6] Given its great distance, the daytime "guest star" observed by the Chinese and Arabs could only have been a supernova—a massive, exploding star, having exhausted its supply of energy from nuclear fusion and collapsed in on itself.

Recent analyses of historical records have found that the supernova that created the Crab Nebula probably occurred in April or early May, rising to its maximum brightness of between apparent magnitude −7 and −4.5 (brighter than everything in the night sky except the Moon) by July. The supernova was visible to the naked eye for about two years after its first observation.^[7] Thanks to the recorded observations of Far Eastern and Middle Eastern astronomers of 1054, Crab Nebula became the first astronomical object recognized as being connected to a supernova explosion.^[6]

iPhone wifi timetrials -- netcat over wifi

I've been doing some housecleaning in my dev-notes and came upon something worthy of a quick paste.
A while back I had noted the performance of the iPhone wifi sitting directly on top of the powerbook-17 1.2ghz running Leopard which was acting as the dedicated AP. I did my best to aim the iPhone at the antenna of the Powerbook on several trial runs to move the speed up or down by a margin of 10-20% by swiveling, rotating, propping, and relocating the iPhone.

Both of the nodes had each other's /etc/hosts sync'd up with the localnetwork default address. There was no default route, nothing but .75 duplex 2.6 ghz and .12 inches between them. (+ a USB cable for sleep prevention)
Netcat is as IO-bound as any program gets, so I am pretty sure this represents the iPhone and powerbook's 802.11g default configuration throughputs.
Scientifically speaking, this is one console-capture, but I ran a dozen partial and full trial runs tests with nearly identical K/s numbers and that was good enough for me.

currently, my iPhone is bone-stock 1.1.3 by accident, awaiting the iPhone SDK. This benchmark took place in December '07.

Setup:

On the iPhone:: socket-connect

# time  nc < /dev/disk0  -vvv   169.254.19.49 9999 

Foehammer.local 

[169.254.19.49] 9999 (distinct) 

open  sent 220610560, rcvd 0  

nc -vvv 169.254.19.49 9999 < /dev/disk0   

0.64s user 56.12s system 35% cpu  2:41.41 total    

On the Powerbook::

jim@Foehammer ~/ipod $ time nc -vvv  -w30 -l  -p 9999  >iphone.disk0.dmg
listening on [any] 9999 ...
connect to [169.254.19.49]
from iPhone-001CB36763B2.local [169.254.222.51] 49519
sent 0, rcvd 220546904

real    2m43.590s
user    0m0.638s
sys    0m9.693s

Google's math of the outcome

((220546904 bytes) / (((120 seconds)) + (43.59000 seconds))) / Mbps = 10.2857103 Mbps

The take-home

The end-goal was determining if Xbox XBMC could hook up to iPhone ftp/samba/upnp shares and play videos without preloading the Xbox hard drive. the answer is most MP4 which fall under 4Mbit, but due to CPU requirements, only "iPod" (768Kb/s) h.264 would be feasable (due to the Xbox's meager 733 MegaHamsterZ).

Higher media throughputs might work if the Xbox sits on the wifi AP connected by cat3 wire or better, and the compression is nil, but in reality mine sits usb-powering its portable d-link DWL-G730AP, and it bogs mightily under the dlink bargain wifi hardware curse.

post-script

The follow-on project to this experiment was to facilitate a python or ruby torrent client intermittently droning along on EDGE bandwidth, staging various file acquisition activities. That particular notion was not a success.
xvid-avis, wmv, etc under 4Mbits would likely work rock steady as well, but chances are they would only be there in the first place in the bittorrent completed files zone, since porting mplayer, vlc, or ffplay was not on any of my iPhone todo's (not prior to a Gentoo install).

The Diamond Age By Neal Stephenson

... and where are we today?

Man I have loved this book since the moment I read it on an 8 hour flight across the country. Any dedicated hacker that catches wind of the prevailing programming concepts[of self-seeding matter and molecules] can't help but begin to consider adopting the perspective that enables self-expanding works from recursive generative algorithms.

The Diamond Age By Neal Stephenson Reviewed at memoirs on a rainy day: "to put together something they call the Seed, a way of being independent of the Feed and liberating his society.

The Seed, an advanced technology that would allow decentralized compilation of matter (as opposed to the centralized pipelines that currently supply basic molecules through the “Feed”). The “Seed” technology would be advantageous for Chinese culture, which is grounded in peasant labor destroyed by Western industrial society..."

Competitive and self-expanding code Seed strategies prevail in Redcode like nowhere else.

In the world of software languages, brainfuck right up there with a potential to first define the primitives and then define the side effects from seed values plugged in to the primitives.

Distilled Network Mobility -- Patterns in Network Architecture: A Return to Fundamentals

I was browsing through a network mobility resource in oreilly safari ...

BOOK

Patterns in Network Architecture: A Return to Fundamentals

By John Day

Table of Contents

... this book describes to a T the fundamentals of a distributed and mobile object namespace though tends to get irate about certain EITF shortcomings...

but the fact is that an IPv4 address with or without the euphamism of DNS is about as mobile as a tree-root. IPV4 traps the essence of distributed and mobile agent systems from all but the very highest layers of abstractions. I'm not sure I ever counted the number of abstractions rolled into one dissertation of a fused enterprise bus/mobile-distributed agent and namespace/high performance supercomputing swarm platform...

...the world of standards operates in concrete and encapsulated terms that number in the dozens when considering the challenges faced to mobilize code (web isn't even in the same ballpark, or more than a transient conduit) on an ipv4 baseline network... over the years I've seen language developing around concepts far too vastly compounded and complex to iterate when i first started employing these concepts. The language I've used to date is typically as terse as is necessary to pitch products with the benefits built in from patterning after the idealized distributed agent models, however only 1 in 1000's of software professionals or funding sources i've met has voiced or acted on any significant understandings and championing of mobile software agents.

...I found an interesting excerpt that describes the engineering process in general, and very strongly describes nearly every project I've contributed innovations into... I'd liken these analogous to metaphors of hunters and farmers.. or perhaps the options seekers vs. the procedural mindsets.

from John Day's book...

'If one looks carefully over this chronology, one is inexorably drawn to the conclusion that the Internet did not begin to stagnate at the turn of the century, but in the late 1970s. It is here that conservative behavior begins to take hold. It is here that each juncture is not seen as an opportunity to fulfill the earlier vision or to synthesize new insights with previous achievements to create a new innovative direction, but more the minimal change is made to solve the immediate problem and go no further. It begins to seem that people are the keepers of some flame and dare not tamper with what has been entrusted to them, the classic behavior of a "second generation" i.e., not the founding generation. Some will argue that this was the policy of small incremental change that has been a stalwart of Internet development. And they are right; caution in maintaining a production system is a necessity but a slow death for research and the research was far from done. But isn't that just what one would expect from the inheritors of the flame? Is that just a rationalization of a much deeper malaise?'

while I'm not really as incensed as John Day about the challenges of ipv4 or the common theme that those who seek job security often shy from loving their work, I do like the blunt picture of futility vetted succinctly and concisely in one chapter of a book underscoring the challenges of building 'timeless' software agent designs in a stunted and self-afflicted problem domain.

• Stateless infrastructure stands a better chance of being flawless software foundation than API counterparts which exist as islands of similar base classes.
  
• Databases are the doorstep of an enterprise, hardly the foundation, and boy can Oracle screw up a simple socket data delivery design...
  
• The business plan is in the hands of its engineers.