[Stgt-devel] SSC, Variable Block Sizes and file marks etc ...

[Mark Harvey]

On Sat, Aug 2, 2008 at 2:39 AM, Richard Sharpe
<realrichardsharpe at gmail.com> wrote:
> Hi,
>
> I discovered Bacula's btape command and started using that to verify
> my SSC implementation, and then discovered that I would have to
> implement variable block size records in any case, so I switched.
>
> I can now pass all the important Bacula btest tests, and many of the
> unimportant ones like back space file, back space record, but there is
> still at least one unimportant test that my code fails (space to just
> past the very last record on the tape, but fixing that required a
> careful reading of the spec ... :-). The code I have at the moment is
> pretty crappy, and I might post it in a couple of weeks when I get
> back from a cruise we are taking next week.
>
> However, I thought I would describe the approach I have taken.
>
> I keep all the metadata separate from the data. I have a separate
> array of filemarks/setmarks (actually, I only implement filemarks at
> the moment, but setmarks are not hard). This array can be small since
> we don't expect many of them, and the cost of realloc to double its
> size from 20 to 40 entries is not expensive and is only needed with
> the WITE FILEMARKS(6) command.
>
> I also have a separate array of record lengths. I size it for 1M
> records at the moment, but the sizing of this array will depend on the
> size of tape you are emulating and the record sizes you expect to
> support. It seems that the default in programs like NetBackup is 64kB,
> but they routinely recommend that you use 512kB or larger. 1M records
> with 1MB record sizes means we can support 1TB backup tapes. Since
> uncompressed LTO4s are 800GB (I am being a little loose here with MB
> and MiB etc), and since compression can be done below the tape level,
> 1M records seems fine and requires 4MB per tape emulated. I would
> probably return 64kB or more as the min Block Size (although I wonder
> how many apps would support a granularity larger than 9 (16, for
> example ...).

NetBackup uses a 1k block for media header and image header information.

On a tape load, the 'bptm' process requests a '64k read'. If it
receives anything other then 1k of data, it rejects the tape as not
being in NetBackups format.

>
> Every write writes its length to the next available record slot.
>
> When reading, we keep track of the current record (starting at 0) and
> read only the bytes of data the current record slot tells us
> (actually, we should read the min of current record and the requested
> size) and then deal with under-length or over-length reads as the spec
> requests.
>
> The filemarks array is used in this way. Marks occur before records.
> If a mark is at the same offset as a record, the mark is first, and I
> keep two variables, next_mark and next_record to tell me what I am up
> to.
>
> So, during a read, if there is a mark at the current offset, then a
> CHECK_CONDITION should be returned with NO_SENSE and ASC_MARK.
> However, if the request was an over-length read and there is a MARK
> immediately after the read, a CHECK_CONDITION with NO_SENSE and NO_ASC
> should be returned (at least by my reading of the spec).
> Unfortunately, there are no programs that I am aware of that test for
> these things (perhaps a useful additional tool for STGT would be a
> program that does these sorts of tests). I am not sure that my code
> handles this condition correctly, yet, as there is no test for it.
>
> Spacing forward is a simple matter of lseeking using the block lengths
> that I keep in my array of record sizes, and spacing backward is much
> the same ...
>
> My code currently does not handle writing over a MARK yet, but that is
> only a matter of time, and I will probably write a stress test tool
> first.
I've yet to find any program that uses setmarks..

>
> Also, I am not currently saving the metadata to files/xattrs nor
> reading them in when a tape if mounted, but that is the easy part.
>
> I have chosen this representation because performance in the data path
> is important to me, especially during writing, but it is clear that
> other representations can be used.
> _______________________________________________
> Stgt-devel mailing list
> Stgt-devel at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/stgt-devel
>

Mark