Hi Casey,
Thanks. I even got a compiling error with fio-objectstore branch.
Here is error message:
make[3]: *** No rule to make target `rbd_fuse/rbd-fuse.c', needed by
`rbd_fuse/rbd-fuse.o'. Stop.
make[3]: *** Waiting for unfinished jobs....
CXX ceph_fuse.o
make[3]: Leaving directory `/home/jamesliu/WorkSpace/ceph_fio/src'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/home/jamesliu/WorkSpace/ceph_fio/src'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/home/jamesliu/WorkSpace/ceph_fio/src'
make: *** [all-recursive] Error 1
jamesliu@jamesliu-OptiPlex-7010:~/WorkSpace/ceph_fio$ git branch
* fio-objectstore
master
Regards,
James
-----Original Message-----
From: Casey Bodley [mailto:[email protected]]
Sent: Thursday, September 03, 2015 10:44 AM
To: James (Fei) Liu-SSI
Cc: [email protected]
Subject: Re: About Fio backend with ObjectStore API
Hi James,
I'm sorry for not following up on that segfault, but I wasn't ever able to
reproduce it. I used it recently for memstore testing without any problems. I
wonder if there's a problem with the autotools build? I've only tested it with
cmake. When I find some time, I'll rebase it on master and do another round of
testing.
Casey
----- Original Message -----
> From: "James (Fei) Liu-SSI" <[email protected]>
> To: "Haomai Wang" <[email protected]>, "Casey Bodley"
> <[email protected]>
> Cc: "Casey Bodley" <[email protected]>, "Matt W. Benjamin"
> <[email protected]>, [email protected]
> Sent: Wednesday, September 2, 2015 8:06:14 PM
> Subject: RE: About Fio backend with ObjectStore API
>
> Hi Haomai and Case,
> Do you have any fixes for that segfault?
>
> Thanks,
> James
>
> -----Original Message-----
> From: Haomai Wang [mailto:[email protected]]
> Sent: Wednesday, July 22, 2015 6:07 PM
> To: Casey Bodley
> Cc: Casey Bodley; Matt W. Benjamin; James (Fei) Liu-SSI;
> [email protected]
> Subject: Re: About Fio backend with ObjectStore API
>
> no special
>
> [global]
> #logging
> #write_iops_log=write_iops_log
> #write_bw_log=write_bw_log
> #write_lat_log=write_lat_log
> ioengine=./ceph-int/src/.libs/libfio_ceph_objectstore.so
> invalidate=0 # mandatory
> rw=write
> #bs=4k
>
> [filestore]
> iodepth=1
> # create a journaled filestore
> objectstore=filestore
> directory=./osd/
> filestore_journal=./osd/journal
>
> On Thu, Jul 23, 2015 at 4:56 AM, Casey Bodley <[email protected]> wrote:
> > Hi Haomai,
> >
> > Sorry for the late response, I was out of the office. I'm afraid I
> > haven't run into that segfault. The io_ops should be set at the very
> > beginning when it calls get_ioengine(). All I can suggest is that
> > you verify that your job file is pointing to the correct
> > fio_ceph_objectstore.so. If you've made any other interesting
> > changes to the job file, could you share it here?
> >
> > Casey
> >
> > ----- Original Message -----
> > From: "Haomai Wang" <[email protected]>
> > To: "Casey Bodley" <[email protected]>
> > Cc: "Matt W. Benjamin" <[email protected]>, "James (Fei) Liu-SSI"
> > <[email protected]>, [email protected]
> > Sent: Tuesday, July 21, 2015 7:50:32 AM
> > Subject: Re: About Fio backend with ObjectStore API
> >
> > Hi Casey,
> >
> > I check your commits and know what you fixed. I cherry-picked your
> > new commits but I still met the same problem.
> >
> > """
> > It's strange that it alwasys hit segment fault when entering
> > "_fio_setup_ceph_filestore_data", gdb tells "td->io_ops" is NULL but
> > when I up the stack, the "td->io_ops" is not null. Maybe it's
> > related to dlopen?
> > """
> >
> > Do you have any hint about this?
> >
> > On Thu, Jul 16, 2015 at 5:23 AM, Casey Bodley <[email protected]> wrote:
> >> Hi Haomai,
> >>
> >> I was able to run this after a couple changes to the filestore.fio
> >> job file. Two of the config options were using the wrong names. I
> >> pushed a fix for the job file, as well as a patch that renames
> >> everything from filestore to objectstore (thanks James), to
> >> https://github.com/linuxbox2/linuxbox-ceph/commits/fio-objectstore.
> >>
> >> I found that the read support doesn't appear to work anymore, so
> >> give "rw=write" a try. And because it does a mkfs(), make sure
> >> you're pointing it to an empty xfs directory with the "directory=" option.
> >>
> >> Casey
> >>
> >> On Tue, Jul 14, 2015 at 2:45 AM, Haomai Wang <[email protected]> wrote:
> >>> Anyone who have successfully ran the fio with this external io
> >>> engine ceph_objectstore?
> >>>
> >>> It's strange that it alwasys hit segment fault when entering
> >>> "_fio_setup_ceph_filestore_data", gdb tells "td->io_ops" is NULL
> >>> but when I up the stack, the "td->io_ops" is not null. Maybe it's
> >>> related to dlopen?
> >>>
> >>> On Fri, Jul 10, 2015 at 3:51 PM, Haomai Wang
> >>> <[email protected]>
> >>> wrote:
> >>>> I have rebased the branch with master, and push it to ceph
> >>>> upstream repo.
> >>>> https://github.com/ceph/ceph/compare/fio-objectstore?expand=1
> >>>>
> >>>> Plz let me know if who is working on this. Otherwise, I would
> >>>> like to improve this to be merge ready.
> >>>>
> >>>> On Fri, Jul 10, 2015 at 4:26 AM, Matt W. Benjamin
> >>>> <[email protected]>
> >>>> wrote:
> >>>>> That makes sense.
> >>>>>
> >>>>> Matt
> >>>>>
> >>>>> ----- "James (Fei) Liu-SSI" <[email protected]> wrote:
> >>>>>
> >>>>>> Hi Casey,
> >>>>>> Got it. I was directed to the old code base. By the way,
> >>>>>> Since the testing case was used to exercise all of object stores.
> >>>>>> Strongly recommend to change the name from
> >>>>>> fio_ceph_filestore.cc to fio_ceph_objectstore.cc . And the code
> >>>>>> in fio_ceph_filestore.cc should be refactored to reflect that
> >>>>>> the whole objectstore will be supported by
> >>>>>> fio_ceph_objectstore.cc. what you think?
> >>>>>>
> >>>>>> Let me know if you need any help from my side.
> >>>>>>
> >>>>>>
> >>>>>> Regards,
> >>>>>> James
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Casey Bodley [mailto:[email protected]]
> >>>>>> Sent: Thursday, July 09, 2015 12:32 PM
> >>>>>> To: James (Fei) Liu-SSI
> >>>>>> Cc: Haomai Wang; [email protected]
> >>>>>> Subject: Re: About Fio backend with ObjectStore API
> >>>>>>
> >>>>>> Hi James,
> >>>>>>
> >>>>>> Are you looking at the code from
> >>>>>> https://github.com/linuxbox2/linuxbox-ceph/tree/fio-objectstore?
> >>>>>> It uses ObjectStore::create() instead of new FileStore(). This
> >>>>>> allows us to exercise all of the object stores with the same code.
> >>>>>>
> >>>>>> Casey
> >>>>>>
> >>>>>> On Thu, Jul 9, 2015 at 2:01 PM, James (Fei) Liu-SSI
> >>>>>> <[email protected]> wrote:
> >>>>>> > Hi Casey,
> >>>>>> > Here is the code in the fio_ceph_filestore.cc. Basically,
> >>>>>> > it
> >>>>>> creates a filestore as backend engine for IO exercises. If we
> >>>>>> got to send IO commands to KeyValue Store or Newstore, we got
> >>>>>> to change the code accordingly, right? I did not see any other
> >>>>>> files like fio_ceph_keyvaluestore.cc or fio_ceph_newstore.cc.
> >>>>>> In my humble opinion, we might need to create other two fio
> >>>>>> engines for keyvaluestore and newstore if we want to exercise
> >>>>>> these two, right?
> >>>>>> >
> >>>>>> > Regards,
> >>>>>> > James
> >>>>>> >
> >>>>>> > static int fio_ceph_filestore_init(struct thread_data *td)
> >>>>>> > 209 {
> >>>>>> > 210 vector<const char*> args;
> >>>>>> > 211 struct ceph_filestore_data *ceph_filestore_data = (struct
> >>>>>> ceph_filestore_data *) td->io_ops->data;
> >>>>>> > 212 ObjectStore::Transaction ft;
> >>>>>> > 213
> >>>>>> >
> >>>>>> > 214 global_init(NULL, args, CEPH_ENTITY_TYPE_OSD,
> >>>>>> CODE_ENVIRONMENT_UTILITY, 0);
> >>>>>> > 215 //g_conf->journal_dio = false;
> >>>>>> > 216 common_init_finish(g_ceph_context);
> >>>>>> > 217 //g_ceph_context->_conf->set_val("debug_filestore", "20");
> >>>>>> > 218 //g_ceph_context->_conf->set_val("debug_throttle", "20");
> >>>>>> > 219 g_ceph_context->_conf->apply_changes(NULL);
> >>>>>> > 220
> >>>>>> >
> >>>>>> > 221 ceph_filestore_data->osd_path =
> >>>>>> strdup("/mnt/fio_ceph_filestore.XXXXXXX");
> >>>>>> > 222 ceph_filestore_data->journal_path =
> >>>>>> strdup("/var/lib/ceph/osd/journal-ram/fio_ceph_filestore.XXXXXXX"
> >>>>>> );
> >>>>>> > 223
> >>>>>> >
> >>>>>> > 224 if (!mkdtemp(ceph_filestore_data->osd_path)) {
> >>>>>> > 225 cout << "mkdtemp failed: " << strerror(errno) <<
> >>>>>> std::endl;
> >>>>>> > 226 return 1;
> >>>>>> > 227 }
> >>>>>> > 228 //mktemp(ceph_filestore_data->journal_path); // NOSPC issue
> >>>>>> > 229
> >>>>>> >
> >>>>>> > 230 ObjectStore *fs = new
> >>>>>> FileStore(ceph_filestore_data->osd_path,
> >>>>>> ceph_filestore_data->journal_path);
> >>>>>> > 231 ceph_filestore_data->fs = fs;
> >>>>>> > 232
> >>>>>> >
> >>>>>> > 233 if (fs->mkfs() < 0) {
> >>>>>> > 234 cout << "mkfs failed" << std::endl;
> >>>>>> > 235 goto failed;
> >>>>>> > 236 }
> >>>>>> > 237
> >>>>>> > 238 if (fs->mount() < 0) {
> >>>>>> > 239 cout << "mount failed" << std::endl;
> >>>>>> > 240 goto failed;
> >>>>>> > 241 }
> >>>>>> > 242
> >>>>>> >
> >>>>>> > 243 ft.create_collection(coll_t());
> >>>>>> > 244 fs->apply_transaction(ft);
> >>>>>> > 245
> >>>>>> >
> >>>>>> > 246
> >>>>>> >
> >>>>>> > 247 return 0;
> >>>>>> > 248
> >>>>>> >
> >>>>>> > 249 failed:
> >>>>>> > 250 return 1;
> >>>>>> > 251
> >>>>>> >
> >>>>>> > 252 }
> >>>>>> > -----Original Message-----
> >>>>>> > From: Casey Bodley [mailto:[email protected]]
> >>>>>> > Sent: Thursday, July 09, 2015 9:19 AM
> >>>>>> > To: James (Fei) Liu-SSI
> >>>>>> > Cc: Haomai Wang; [email protected]
> >>>>>> > Subject: Re: About Fio backend with ObjectStore API
> >>>>>> >
> >>>>>> > Hi James,
> >>>>>> >
> >>>>>> > In the job file src/test/filestore.fio, you can modify the
> >>>>>> > line "objectstore=filestore" to use any objectstore type
> >>>>>> > supported by
> >>>>>> the
> >>>>>> > ObjectStore::create() factory.
> >>>>>> >
> >>>>>> > Casey
> >>>>>> >
> >>>>>> > On Wed, Jul 8, 2015 at 8:02 PM, James (Fei) Liu-SSI
> >>>>>> <[email protected]> wrote:
> >>>>>> >> Hi Casey,
> >>>>>> >> Quick questions, The code in the trunk only cover the test
> >>>>>> >> for
> >>>>>> filestore. I was wondering do you have any plan to cover the
> >>>>>> test for kvstore and newstore?
> >>>>>> >>
> >>>>>> >> Thanks,
> >>>>>> >> James
> >>>>>> >>
> >>>>>> >> -----Original Message-----
> >>>>>> >> From: [email protected]
> >>>>>> >> [mailto:[email protected]] On Behalf Of James
> >>>>>> >> (Fei)
> >>>>>>
> >>>>>> >> Liu-SSI
> >>>>>> >> Sent: Tuesday, June 30, 2015 2:19 PM
> >>>>>> >> To: Casey Bodley
> >>>>>> >> Cc: Haomai Wang; [email protected]
> >>>>>> >> Subject: RE: About Fio backend with ObjectStore API
> >>>>>> >>
> >>>>>> >> Hi Casey,
> >>>>>> >>
> >>>>>> >> Thanks a lot.
> >>>>>> >>
> >>>>>> >> Regards,
> >>>>>> >> James
> >>>>>> >>
> >>>>>> >> -----Original Message-----
> >>>>>> >> From: Casey Bodley [mailto:[email protected]]
> >>>>>> >> Sent: Tuesday, June 30, 2015 2:16 PM
> >>>>>> >> To: James (Fei) Liu-SSI
> >>>>>> >> Cc: Haomai Wang; [email protected]
> >>>>>> >> Subject: Re: About Fio backend with ObjectStore API
> >>>>>> >>
> >>>>>> >> Hi,
> >>>>>> >>
> >>>>>> >> When Danny Al-Gaaf & Daniel Gollub published "Ceph
> >>>>>> >> Performance
> >>>>>> >> Analysis: fio and RBD" at
> >>>>>> >>
> >>>>>> https://telekomcloud.github.io/ceph/2014/02/26/ceph-performance
> >>>>>> -a
> >>>>>> naly
> >>>>>> >> s is_fio_rbd.html, they also mentioned a fio engine that
> >>>>>> >> linked directly into ceph's FileStore. I was able to find
> >>>>>> >> Daniel's branch
> >>>>>> on
> >>>>>> >> github at
> >>>>>> >> https://github.com/gollub/ceph/tree/fio_filestore_v2, and
> >>>>>> did some more work on it at the time.
> >>>>>> >>
> >>>>>> >> I just rebased that work onto the latest ceph master branch,
> >>>>>> >> and
> >>>>>> pushed to our github at
> >>>>>> https://github.com/linuxbox2/linuxbox-ceph/tree/fio-objectstore.
> >>>>>> You can find the source in src/test/fio_ceph_filestore.cc, and
> >>>>>> run fio with the provided example fio job file in
> >>>>>> src/test/filestore.fio.
> >>>>>> >>
> >>>>>> >> I didn't have a chance to confirm that it builds with
> >>>>>> >> automake, but
> >>>>>>
> >>>>>> >> the cmake version built for me. I'm happy to help if you run
> >>>>>> >> into problems, Casey
> >>>>>> >>
> >>>>>> >> On Tue, Jun 30, 2015 at 2:31 PM, James (Fei) Liu-SSI
> >>>>>> <[email protected]> wrote:
> >>>>>> >>> Hi Haomai,
> >>>>>> >>> What are you trying to ask is to benchmark local
> >>>>>> objectstore(like kvstore/filestore/newstore) locally with
> >>>>>> FIO(ObjectStore engine)? You want to purely compare the
> >>>>>> performance locally for these objectstores, right?
> >>>>>> >>>
> >>>>>> >>> Regards,
> >>>>>> >>> James
> >>>>>> >>>
> >>>>>> >>> -----Original Message-----
> >>>>>> >>> From: [email protected]
> >>>>>> >>> [mailto:[email protected]] On Behalf Of
> >>>>>> >>> Haomai
> >>>>>> Wang
> >>>>>> >>> Sent: Tuesday, June 30, 2015 9:06 AM
> >>>>>> >>> To: [email protected]
> >>>>>> >>> Subject: About Fio backend with ObjectStore API
> >>>>>> >>>
> >>>>>> >>> Hi all,
> >>>>>> >>>
> >>>>>> >>> Long long ago, is there someone said about fio backend with
> >>>>>> >>> Ceph
> >>>>>> ObjectStore API? So we could use the existing mature fio
> >>>>>> facility to benchmark ceph objectstore.
> >>>>>> >>>
> >>>>>> >>> --
> >>>>>> >>> Best Regards,
> >>>>>> >>>
> >>>>>> >>> Wheat
> >>>>>> >>> --
> >>>>>> >>> To unsubscribe from this list: send the line "unsubscribe
> >>>>>> ceph-devel"
> >>>>>> >>> in the body of a message to [email protected] More
> >>>>>> majordomo
> >>>>>> >>> info at http://vger.kernel.org/majordomo-info.html
> >>>>>> >> {.n + +% lzwm b 맲 r yǩ ׯzX ܨ} Ơz &j:+v zZ+
> >>>>>> +zf h ~ i z w ? & )ߢ f
> >>>>>> N r y b X ǧv ^ ){.n + z ]z {ay ʇڙ ,j f h z w
> >>>>>> j:+v w j m zZ+ ݢj"
> >>>>>
> >>>>> --
> >>>>> Matt Benjamin
> >>>>> CohortFS, LLC.
> >>>>> 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103
> >>>>>
> >>>>> http://cohortfs.com
> >>>>>
> >>>>> tel. 734-761-4689
> >>>>> fax. 734-769-8938
> >>>>> cel. 734-216-5309
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Best Regards,
> >>>>
> >>>> Wheat
> >>>
> >>>
> >>>
> >>> --
> >>> Best Regards,
> >>>
> >>> Wheat
> >
> >
> >
> > --
> > Best Regards,
> >
> > Wheat
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > in the body of a message to [email protected] More majordomo
> > info at http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> Best Regards,
>
> Wheat
> N r y b X ǧv ^ ){.n + z ]z {ay ʇڙ ,j f h z w j
> :+v w j m zZ+ ݢj"
