On Sat, Aug 2, 2008 at 2:39 AM, Richard Sharpe <realrichardsharpe at gmail.com> wrote: > Hi, > > I discovered Bacula's btape command and started using that to verify > my SSC implementation, and then discovered that I would have to > implement variable block size records in any case, so I switched. > > I can now pass all the important Bacula btest tests, and many of the > unimportant ones like back space file, back space record, but there is > still at least one unimportant test that my code fails (space to just > past the very last record on the tape, but fixing that required a > careful reading of the spec ... :-). The code I have at the moment is > pretty crappy, and I might post it in a couple of weeks when I get > back from a cruise we are taking next week. > > However, I thought I would describe the approach I have taken. > > I keep all the metadata separate from the data. I have a separate > array of filemarks/setmarks (actually, I only implement filemarks at > the moment, but setmarks are not hard). This array can be small since > we don't expect many of them, and the cost of realloc to double its > size from 20 to 40 entries is not expensive and is only needed with > the WITE FILEMARKS(6) command. > > I also have a separate array of record lengths. I size it for 1M > records at the moment, but the sizing of this array will depend on the > size of tape you are emulating and the record sizes you expect to > support. It seems that the default in programs like NetBackup is 64kB, > but they routinely recommend that you use 512kB or larger. 1M records > with 1MB record sizes means we can support 1TB backup tapes. Since > uncompressed LTO4s are 800GB (I am being a little loose here with MB > and MiB etc), and since compression can be done below the tape level, > 1M records seems fine and requires 4MB per tape emulated. I would > probably return 64kB or more as the min Block Size (although I wonder > how many apps would support a granularity larger than 9 (16, for > example ...). NetBackup uses a 1k block for media header and image header information. On a tape load, the 'bptm' process requests a '64k read'. If it receives anything other then 1k of data, it rejects the tape as not being in NetBackups format. > > Every write writes its length to the next available record slot. > > When reading, we keep track of the current record (starting at 0) and > read only the bytes of data the current record slot tells us > (actually, we should read the min of current record and the requested > size) and then deal with under-length or over-length reads as the spec > requests. > > The filemarks array is used in this way. Marks occur before records. > If a mark is at the same offset as a record, the mark is first, and I > keep two variables, next_mark and next_record to tell me what I am up > to. > > So, during a read, if there is a mark at the current offset, then a > CHECK_CONDITION should be returned with NO_SENSE and ASC_MARK. > However, if the request was an over-length read and there is a MARK > immediately after the read, a CHECK_CONDITION with NO_SENSE and NO_ASC > should be returned (at least by my reading of the spec). > Unfortunately, there are no programs that I am aware of that test for > these things (perhaps a useful additional tool for STGT would be a > program that does these sorts of tests). I am not sure that my code > handles this condition correctly, yet, as there is no test for it. > > Spacing forward is a simple matter of lseeking using the block lengths > that I keep in my array of record sizes, and spacing backward is much > the same ... > > My code currently does not handle writing over a MARK yet, but that is > only a matter of time, and I will probably write a stress test tool > first. I've yet to find any program that uses setmarks.. > > Also, I am not currently saving the metadata to files/xattrs nor > reading them in when a tape if mounted, but that is the easy part. > > I have chosen this representation because performance in the data path > is important to me, especially during writing, but it is clear that > other representations can be used. > _______________________________________________ > Stgt-devel mailing list > Stgt-devel at lists.berlios.de > https://lists.berlios.de/mailman/listinfo/stgt-devel > Mark |