[stgt] [PATCH 0/2] RFC - Double-linked list backing store.

[Mark Harvey]

On Thu, Oct 2, 2008 at 10:27 AM, FUJITA Tomonori
<fujita.tomonori at lab.ntt.co.jp> wrote:
> On Thu, 2 Oct 2008 09:42:12 +1000
> "Mark Harvey" <markh794 at gmail.com> wrote:
>
>> On Thu, Oct 2, 2008 at 8:55 AM, FUJITA Tomonori
>> <fujita.tomonori at lab.ntt.co.jp> wrote:
>> > On Thu, 2 Oct 2008 08:41:55 +1000
>> > "Mark Harvey" <markh794 at gmail.com> wrote:
>> >
>> >> On Wed, Oct 1, 2008 at 10:04 PM, FUJITA Tomonori
>> >> <fujita.tomonori at lab.ntt.co.jp> wrote:
>> >> > On Mon, 18 Aug 2008 15:47:00 +1000
>> >> > Mark Harvey <markh794 at gmail.com> wrote:
>> >> >
>> >> >> An alternative backing store primarily for SSC device which uses a meta-data header for each block of data.
>> >> >>
>> >> >> Embedded with the metadata are forward & back pointers to next & previous headers forming a double-linked list of headers.
>> >> >>
>> >> >> Patch 1/2 patch includes bs_tape and a couple of helper utilities for:
>> >> >> - creating 'blank' media
>> >> >> - Dumping media headers
>> >> >> Patch 2/2 patches the tgt-core-test script which I use to setup the VTL.
>> >> >>
>> >> >> Note: The bs_tape is almost at the same level of functionality as the bs_ssc without Richard Sharps recent patches.
>> >> >>
>> >> >>
>> >> >> Pros: (I can see) of the embedded metadata headers include the ability to store variable data sizes (compressed, encrypted or just variable block writes). Metadata headers do not have to contain data (FILEMARKS, SETMARKS, End of Data markers etc). Not limited by preallocated structure at media creation time.
>> >> >>
>> >> >> Cons: Custom format (i.e. can't use with ISO images)
>> >> >
>> >> > Sorry for the late response, I finally read both your and Richard's
>> >> > patches. I'm not sure yet we should store data and metada in a single
>> >> > file (as your patch) or have a separate metadata (as Richard's one).
>> >> >
>> >> > Your file format is something like:
>> >> >
>> >> > blk_header + fixed-size data + blk_header + fixed-size data ...
>> >> >
>> >> >
>> >> > What's your plan about how to handle the variable-block transfers?
>> >> >
>> >> The actual data 'block size' is stored within each blocks blk_header.
>> >> Both original size + size stored on disk is saved as they may differ
>> >> due to backing store applying compression/encryption.
>> >
>> > Yeah, I know that. But I'm not still not sure how we can handle the
>> > variable-block transfers quickly with the format.
>>
>> The format has a narrow focus - to emulate a tape. (sequential
>> access). Seeking will be slower (compared to Richards) as it has to
>
> Yeah, I'm a bit concerned that the format exactly emulates what a tape
> is. We can access to the format only in a sequential way. I thought
> that VTL products use kinda metadata to avoid that. The simplest
> metadata could just save blk_header in sequence.
>
> But I'm not concerned so much. Even if we always access to the format
> in a sequential way, a disk drive is much faster than a tape drive.
> I'm ok with both formats.
>
>
> Can you tell me how your format works with variable block size? The
> file could be something:
>
> blk_header + 56KB data + blk_header + 18KB data + blk_header + 132KB
> data ...

NetBackup uses three different blk sizes..

[Beginning of tape] [1k] [filemark] [64k][64k][64k][<64k][filemark]
(where last block of data is something less then '64k' block size.
Obviously this 64k block size is tunable..

But each blk_header defines the size of the block of data associated.

>
> Or we save blk_header and the fixed-size data?
>
> I don't know anything about SSC (I'm trying to learn it now). Can you
> give me an example how a WRITE command updates the format with
> variable transfers?
>
Sequential device writes never 'replace' data - data is always
appended to the end. Or to put it another way : the end of the last
write is the end of the valid data.

Hence a write command will lay down :
blk_header + data + blk_header(END-OF-DATA HEADER).

The next write will overwrite/update the END-OF-DATA header as a valid
datablock header + data and finally another END-OF-DATA HEADER.

A request to read beyond a 'END-OF-DATA' header results in blank-check
sense code.

Cheers
--
To unsubscribe from this list: send the line "unsubscribe stgt" in
the body of a message to majordomo at vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html