java - Extra EFBFBD bytes in Hadoop thriftfs reading -

-

in hadoop-0.20 have thriftfs contrib, allow access hdfs in other programming language. hadoop provides hdfs.py script demonstration. problem located in do_get , do_put methods.

if use get download utf-8 text file, it's totally ok, when get file in other encoding, can not original file, downloaded file has many "efbfbd" bytes. guess these java codes on hadoopthriftserver may cause problems.

public string read(thrifthandle tout, long offset,                     int length) throws thriftioexception {    try {      = now();      hadoopthrifthandler.log.debug("read: " + tout.id +                                   " offset: " + offset +                                   " length: " + length);      fsdatainputstream in = (fsdatainputstream)lookup(tout.id);      if (in.getpos() != offset) {        in.seek(offset);      }      byte[] tmp = new byte[length];      int numbytes = in.read(offset, tmp, 0, length);      hadoopthrifthandler.log.debug("read done: " + tout.id);      return new string(tmp, 0, numbytes, "utf-8");    } catch (ioexception e) {      throw new thriftioexception(e.getmessage());    }  }

the python code in hdfs.py is

output = open(local, 'wb') path = pathname(); path.pathname = hdfs; input = self.client.open(path)  # find size of hdfs file filesize = self.client.stat(path).length  # read 1mb bytes @ time hdfs offset = 0 chunksize = 1024 * 1024 while true:    chunk = self.client.read(input, offset, chunksize)    if not chunk: break    output.write(chunk)    offset += chunksize    if (offset >= filesize): break  self.client.close(input) output.close()

hope can me.
thanks.


Comments

Post a Comment

Popular posts from this blog

How to access named pipes using JavaScript in Firefox add-on? -

-
i'm trying access named pipe in firefox add-on. code, based on solution 2 this codeproject question, is: var file = components.classes["@mozilla.org/file/local;1"].createinstance(components.interfaces.nsilocalfile); file.initwithpath("\\\\.\\pipe\\test"); var text = "some text written"; var writer = components.classes["@mozilla.org/network/file-output-stream;1"].createinstance(components.interfaces.nsifileoutputstream); // open file read/write access, , create if not exist. writer.init (file, 0x04 | 0x08, -1, 0); writer.write (text, text.length); writer.flush (); writer.close (); when run firefox scratchpad, get: /* exception: component returned failure code: 0x80520012 (ns_error_file_not_found) [nsifileoutputstream.init] @6 */ line 6 line call writer.init. i've played passing different flags writer.init, no luck. i'm able write normal file path code, not named pipe path. i've been searching more information of da...
Read more

multithreading - OPAL (Open Phone Abstraction Library) Transport not terminated when reattaching thread? -

-
i in process of writing application using opal makes h323 calls. have looked online number of working examples , have managed put resembles should like, @ present able call external ip via application when accept call craps out , dies. leaving me with: 0:12.949 setupcall:8752 sert.cxx(259) assertion fail: transport not terminated when reattaching thread , file d:\voip\software\opal\src\opal\transports.cxx, line 1021 passertfunc(0xde3f88, 0xffffffffd228226f, 0, 0) + 0x82 passertfunc(0x10d6e5f8, 0x3fd, 0, 0x10d6eb98) + 0x15b opaltransport::attachthread(0xde5870, 0xffffffffd22fc76b, 0, 0) + 0x96 h323connection::setupconnection(0xffffffffd22fc713, 0, 0, 0xde6ea8) + 0x196 asynchcallsetup(0x10d3572c, 0, 0, 0xdd0108) + 0x7c pthread1arg<psafeptr<opalconnection,psafeptrbase> >::main(0xffffffffd2282faf from have deduced, perhaps incorrectly if there thread lock issue caused (possibly arising fact application sending expected h323 call, throwing in c...
Read more

node.js - req param returns an empty array -

-
i using node.js , mongodb geolocation app , req param returns empty array exports.findpressure = function(req, res) { var queryobject = req.param('q'); console.log(queryobject); db.collection('pressure', function(err, collection) { collection.find( { loc: { $near :[ req.param('q') ] , $maxdistance : 5 }},{"value" : 1, _id : 0}) .sort({_id : -1}).limit(1).toarray(function(err, items) { res.send(items); }); }); }; the longitude , latitude values displayed in console listening on port 3000... connected 'weather' database 8.9068256,52.019347499999995 /pressure?q=8.9068256,52.019347499999995 200 27ms - 2b the url follows http://localhost:3000/pressure?q=8.9068256,52.019347499999995 if use values 8.9068256,52.019347499999995 getting value database if use req.param('q') returning empty array you seem passing values part of query string. req.query.q should contain comma-s...
Read more