Guest Post: Google Takes on Privacy Issues by Allowing ?Opt Outs?

Source: http://feedproxy.google.com/~r/MikazoTechBlog/~3/x8YQ3t_0AhQ/guest-post-google-takes-on-privacy.html

computer repair kansas city missouri dell computer repair kansas city computer repair kansas city kansas computer repair kansas city wornall

Google Now can now be ported to Ice Cream Sandwich

Google Now is one of the coolest new features that Google has brought to Android 4.1 Jelly Bean, but since Jelly Bean has only been announced for three different devices so far, it could be a while before most Android users actually get to use it. But thankfully, an�XDA-Developers forum member has come up with a way to bring Google Now to a few extra users by concocting a port over to Ice Cream Sandwich devices. XDA-Developers’ Will Verduzco writes on the official XDA blog that “by simply modifying the�build.prop” on the app, Ice Cream Sandwich devices can gain access to the program. XDA developer febycv has posted full instructions for completing the port on the XDA Developers forum. For

Source: http://feedproxy.google.com/~r/TheBoyGeniusReport/~3/_ratqYfjh94/

apple computer repair kansas city computer repair kansas city area computer repair kansas city waldo computer repair kansas city

Multi-Post Released

To simplify the process of updating your social network(s) we now offer multi-post in the Chamber Nation system. So now your important system updates like events will be posted with one click to the Chamber Social Network, Facebook and Twitter.

Read more about the Chamber Social Network at: Source: http://ChamberNation.com/blog/2009/10/01/multi-post-released

computer repair overland park ks computer repair overland park kansas computer repair overland park 135th dell computer repair overland park ks

AT&T Unveiling New Blocking Service for Stolen Phones Next Week

Originally announced this past April, AT&T will unveil a database of stolen phones to prevent lost devices from connecting to the carrier's network. The Verge reported the launch with a leaked internal AT&T screenshot showing a launch date of July 10, 2012.


AT&T issued this statement to MacRumors:

As announced in April, AT&T is creating a stolen phone database to prevent devices reported stolen from accessing wireless networks. We will install this availability next week for AT&T phones on our network and are working toward a cross-carrier solution later this year.

The Verge notes that AT&T's guidance for staffers says "only the person who originated a block may request block removal". It's unclear how exactly the database will work, including preventing legitimate transfers in ownership from being flagged by malicious actors, but we expect to have more details from AT&T as the launch date for the stolen phone database comes closer.

Source: http://www.macrumors.com/2012/07/06/att-unveiling-new-blocking-service-for-stolen-phones-next-week/

computer repair kansas city area computer repair kansas city overland park computer repair kansas city mo computer repair kansas city ks

TheLedger.Com Article - Haines City Florida

READ ORIGINAL STORY AT THIS LOCATION:
http://www.theledger.com/article/20100618/NEWS/100619792?p=all&tc=pgall [...]

Source: http://ChamberNation.com/blog/2010/06/18/theledger-com-article-haines-city-florida

computer repair shops computer repair kansas city computer repair kansas city mo computer repair kansas city north

Of course Amazon is building a phone ? but when will it get into the prepaid carrier biz?

Once again, there’s word that Amazon is working on a smartphone, this time from Bloomberg. And given the success of the Kindle Fire so far, which has been the only Android tablet worth buying since its debut (the Nexus 7 …

Source: http://feeds.venturebeat.com/~r/Venturebeat/~3/8yQGkD4HK4E/

computer repair overland park computer repair overland park ks computer repair overland park yelp computer repair overland park ks

Great article on Joining a Chamber of Commerce

Once you select this link be sure to go to PAGE 6

http://www.eq-mag.com/ezine/2010/11 [...]

Source: http://ChamberNation.com/blog/2010/11/12/great-article-on-joining-a-chamber-of-commerce

computer repair overland park kansas computer repair overland park 135th dell computer repair overland park ks computer repair olathe ks

Cisco backpedals after uproar, drops cloud from default router setting

Cisco: Our cloud isn?t for snooping on customers? porn habits.

Source: http://feeds.arstechnica.com/~r/arstechnica/index/~3/YD1_D-xBdsc/

computer repair overland park ks computer repair shawnee ks computer repair services computer repair technician

New Windows Server 2012 puts virtualization front and center

Simpler pricing and fewer SKUs, but no more Small Business Server.

Source: http://feeds.arstechnica.com/~r/arstechnica/index/~3/sBbFt5_d8q0/

computer repair technician computer repair software computer repair tools computer repair training

The dark side of Hadoop

At BackType, we are heavy users of Hadoop. We use it to run computations on our 30TB datastore of social data. We've even open-sourced some significant projects that are built on top of Hadoop.

Unfortunately, Hadoop has problems. It's sloppily implemented and requires all sorts of arcane knowledge to operate it. We would be the first to try out a replacement for Hadoop if a viable alternative existed. In this post, we'll look at some of the darker aspects of Hadoop.

Critical configuration poorly documented

There's a configuration property for Hadoop called "dfs.datanode.max.xcievers". By default, it's set to 256. It turns out if you don't raise this value to be significantly higher (e.g., 5000), your cluster blows up with all sorts of weird errors like EOFExceptions.

As you can imagine, tracking this one down is quite an adventure. Even though this property is so critical, it's not documented anywhere. And on top of that, the property isn't even spelled correctly. It's supposed to be spelled "xceiver" (which is how it's spelled in the source code). See this post for more details.

Terrible with memory usage

We used to have problems with Hadoop running out of memory in various contexts. Sometimes our tasks would randomly die with out of memory exceptions, and sometimes reducers would error during the shuffle phase with "Cannot allocate memory" errors. These errors didn't make a lot of sense to us, since the memory we allocated for the TaskTrackers, DataNodes, and tasks was well under the total memory for each machine.

We traced the problem to a sloppy implementation detail of Hadoop. It turns out that Hadoop sometimes shells out to do various things with the local filesystem. When you shell out in Java, the process gets forked. Forking a process causes the child process to reserve the same amount of memory for itself as the parent process is using (to fully understand what's happening, you need to learn about memory overcommit and the copy-on-write semantics of forking in Linux).This means that the Hadoop process which was using 1GB will temporarily "use" 2GB when it shells out.

The solution to these memory problems is to allocate a healthy amount of swap space for each machine to protect you from these memory glitches. We couldn't believe how much more stable everything became when we added swap space to our worker machines.

Zombies

Hadoop is terrible at process management. It sometimes fails to kill processes that it launches, leaving "zombie tasks" throughout the cluster. These zombie tasks soak up memory, causing out of memory errors for real tasks. We've seen clusters get completely overriden by zombies and become useless.

The problem is that Hadoop puts the burden of a task exiting on the task itself, rather than the TaskTracker that launched it. So if a task enters a zombie state for one reason or another, there's nothing that will clean it up. This is a terrible design. The burden should be on the TaskTracker to supervise the processes it launches and kill them when necessary. Hadoop should not trust user code in tasks to behave properly. This ensures that if I mess up my code, my job may have problems but at least I won't damage the cluster.

A task turning into a zombie seems to be related to encountering an out of memory error, as we've noticed much fewer zombies since fixing Hadoop's other memory issues by adding swap space. We just wish that Hadoop was designed correctly so that zombies were not even possible.

Conclusion

Hadoop continuously finds new and creative ways to frustrate us. Making Hadoop easy to deploy, use, and operate should be the #1 priority for the developers of Hadoop. Hadoop could be an amazing project. Right now though, it's just plain sloppy.

Follow the BackType tech team on Twitter here.

 

Permalink | Leave a comment  »

Source: http://tech.backtype.com/the-dark-side-of-hadoop

computer repair kansas city mo computer repair kansas city ks computer repair kansas city north computer repair kansas city missouri

Apple-Motorola Judge Questions Need for Software Patents

Late last month, federal judge Richard Posner threw out one of the major U.S. cases in the ongoing patent battle between Apple and Motorola, and Reuters today publishes an interesting interview with Posner in which he discusses his view that patents have become too widely used and suggests that there may not be a need for software patents at all.

Noting his belief that software and other industries do not require the same level of patent protection as industries like pharmaceuticals where hundreds of millions of dollars are spent to develop a single protected product, Posner indicates individual software advances require much less economic investment and much of the benefit is gained simply by being first to market.

"It's not clear that we really need patents in most industries," he said.

Also, devices like smartphones have thousands of component features, and they all receive legal protection.

"You just have this proliferation of patents," Posner said. "It's a problem."

In Posner's ruling last month, he noted that Apple's patent on smooth operation of streaming video was in no way a monopoly on all streaming video and that barring an entire product over a single feature would be harmful to consumers. Posner also ruled against Motorola in its efforts to ban the iPhone over standards-essential patents that were to be licensed under fair, reasonable and non-discriminatory (FRAND) terms.


Apple is of course involved in intellectual property disputes with a number of companies, with the cases including both software patents and design rights. Just this week, a ban on U.S. sales of the Samsung Galaxy Tab 10.1 and Galaxy Nexus went into effect as Apple won preliminary injunctions, but the company also experienced setbacks in its battle with HTC in recent days. In those cases, both the U.S. International Trade Commission and a UK court ruled in HTC's favor, with the UK judge ruling that several of Apple's patents including one covering the "slide-to-unlock" feature are invalid in that country.

Source: http://www.macrumors.com/2012/07/05/apple-motorola-judge-questions-need-for-software-patents/

computer repair overland park 135th dell computer repair overland park ks computer repair overland park ks computer repair overland park on 119th

Nanny tutorial by example

Nanny is a language-agnostic tool for managing dependencies between your projects. We use Nanny at BackType to manage our internal Java, Clojure, and Python dependencies. Here's a step-by-step tutorial to show you how to use Nanny:

First, you'll want to get Nanny installed and a repository created on a server somewhere. Follow the installation instructions in the README (should only take a few minutes).

Let's create a project and manage it through Nanny:

mkdir /tmp/project1cd /tmp/project1echo "hello world" > hello.txtecho project1 > CHILDecho "cp *.txt \$1" > CHILDMAKERchmod a+x CHILDMAKERnanny push 1.0.0 "first version of project1"

Now, let's create a new project that depends on project1:

mkdir /tmp/project2cd /tmp/project2echo project1 > NANNYnanny deps

If you look in the _deps/project1 folder, you'll see hello.txt there. Notice that _deps/project1 is a symlink to _deps/_actual/project1-1.0.0. You can look in the _deps/_actual folder to see the currently checked out versions of the dependencies.

Let's go back to project1 and create a new version:

cd /tmp/project1echo "HELLO WORLD" > hello.txtnanny push 0.0.9 "more emphatic hello world"

Uh oh, you'll see an error because you can't push an older version to the repository. Let's try again:

nanny push 1.0.1 "more emphatic hello world"

That should work fine. To see the history of a package, use the "history" command:

nanny history project1

This should print out the following:

1.0.1: more emphatic hello world1.0.0: first version of project1

To see more information about a specific version, use the "info" command:

nanny info project1 1.0.1

This prints out:

--------------------------------------------------------------------------Version control logs:--------------------------------------------------------------------------Message: more emphatic hello world--------------------------------------------------------------------------

Nanny automatically tags each version you push with the first 100 lines of "git log" or "svn log". Since these examples aren't version controlled, you won't see any information in the "Version control logs" section.

Let's go back to project2 and pull in the new dependencies:

cd /tmp/project2nanny depscat _deps/project1/hello.txt

This will print out "HELLO WORLD". If you decide you want the older version of project1, you can specify your dependency in the NANNY file with a specific version, i.e.:

echo "project1 1.0.0" > NANNYnanny depscat _deps/project1/hello.txt

This will print out "hello world".

Let's see how Nanny handles transitive dependencies:

echo "lalala" > project2.txtecho project2 > CHILDecho "cp *.txt \$1" > CHILDMAKERchmod a+x CHILDMAKERnanny push 1.0.0 "first version of project2"mkdir /tmp/project3cd /tmp/project3echo "project2" > NANNYnanny deps

This will pull in the latest version of project2 and all dependencies of project2. Dependency resolution continues transitively (dependencies of your dependencies, dependencies of those dependencies, etc.)

Let's see what happens when there's a conflict:

echo "project1project2" > NANNYnanny deps

project2 depends on v1.0.0 of project1, but project3's NANNY file states that it wants the latest version of project 1 (v1.0.1). Nanny resolves conflicts by choosing the latest version, so in this case it chooses v1.0.1 of project1.

That's really all there is to it. As we've seen, all dependency management occurs through three files:

• NANNY - specify dependencies of this project
• CHILD - name the packages created by this project to be used as dependencies by other projects
• CHILDMAKER - a script to package up this project into a dependency

Be sure to check out the README file in the project for more information. You can get more information on all the commands by executing "nanny help". Enjoy!

You can follow me on Twitter at @nathanmarz.

Permalink | Leave a comment  »

Source: http://tech.backtype.com/nanny-tutorial-by-example

dell computer repair overland park ks computer repair olathe ks computer repair kansas city computer repair lawrence ks

How To Add Audio Streaming To Your Website

Source: http://feedproxy.google.com/~r/MikazoTechBlog/~3/y_h0wvu717s/how-to-add-audio-streaming-to-your_06.html

computer repair kansas city computer repair lawrence ks computer repair overland park computer repair blue springs mo

Apple Facing Potential Suspension of Italian Sales Amid Warranty Concerns

Late last year, Apple was fined $1.2 million by Italian regulators over warranty-related issues, with Apple being cited for not adequately disclosing standard two-year warranties required to be offered with new products sold in the country.

Following the ruling, Apple temporarily added disclosures to its Italian online store even as it was appealing the decision. Apple officially lost its appeal earlier this year, with the company being required to pay the previously-assessed fine.

Reuters now reports that Italian regulators remain concerned about Apple's sales and marketing tactics and are considering additional fines and an eventual temporary ban on Apple's ability to sell products in the country as further sanctions.

The AGCM said in its monthly bulletin that Apple was continuing to adopt unfair commercial practices in Italy and noted this could eventually lead to the closure of its Italian operations for up to 30 days.

The U.S. group has 30 days to respond.

The AGCM alleges that information provided by Apple about an extra guarantee scheme encourages customers to buy the service without clearly explaining that the company is obliged to offer a two-year free warranty, the source said.

Italy is not the only country where Apple is facing criticism over its warranty policies. Earlier this year, consumer groups in eleven European Union countries filed letters of complaint with their respective regulatory agencies asking them to take action against Apple's misleading warranty sales tactics in light of the EU's requirement for a standard two-year warranty. The company clarified the differences between EU protection and Apple's warranty policies, but concerns about its disclosures apparently still remain.

Update: Apple takes issue with the Italian antitrust authority, saying in a statement:

We have introduced a number of measures to address the Italian competition authority concerns and we disagree with their latest complaint.

Source: http://www.macrumors.com/2012/07/02/apple-facing-potential-suspension-of-italian-sales-amid-warranty-concerns/

computer repair technician computer repair software computer repair tools computer repair training

Traffic and navigation app Waze drives past 20 million users

People-powered traffic and navigation application Waze is mightier than ever after having doubled in size in six months. Today, the four-year-old company celebrates 20 million members who’ve accumulated more than 3.2 billion miles of on-the-road time with Waze in tow.…

Source: http://feeds.venturebeat.com/~r/Venturebeat/~3/BcUYt_75hZI/

computer repair kansas city waldo computer repair kansas city computer repair kansas city wornall computer repair kansas city ks

Chamber Social Network

Chambers of Commerce are realizing the value in social networking, but most Chambers have not seen the success they had wished for after they launced a group or even a complete platform. Chamber members want business and anything they lend their time too that will help them grow they're interested in. We are pleased to announce the first of [...]

Source: http://ChamberNation.com/blog/2009/06/29/chamber-social-network

computer repair technician computer repair software computer repair tools computer repair training

Preview of Storm: The Hadoop of Realtime Processing

We have been doing realtime processing for a long time at BackType. We've recently developed a new system for doing realtime processing called Storm to replace our old system of queues and workers. Storm is a distributed, reliable, and fault-tolerant stream processing system. Its use cases are so broad that we consider it to be a fundamental new primitive for data processing. That's why we call it the Hadoop of realtime: it does for realtime processing what Hadoop does for batch processing. We are planning to open-source Storm sometime in the next few months.

Like most people doing realtime processing, our old system was a complicated graph of queues and workers. Worker processes would take messages off a queue and then update a database and/or fire off new messages to other queues. 

There was a lot of pain in doing realtime processing this way. We found that we spent most of our time worrying about where to send messages, where to receive messages from, deploying workers, and deploying queues. Worst of all, the system wasn't fault tolerant: we had to make sure our queues and workers stayed up.

Storm solves these issues completely. It abstracts the message passing away, automatically parallelizes the stream computation on a cluster of machines, and lets you focus on your realtime processing logic. Even more interesting, Storm enables a whole new range of applications we didn't anticipate when we initially designed it.

Properties of Storm

Here are the key properties of Storm:

1. Simple programming model: Just like how MapReduce dramatically lowers the complexity for doing parallel batch processing, Storm's programming model dramatically lowers the complexity for doing realtime processing. 

2. Runs any programming language: Even though Storm runs on the JVM (and is written in Clojure), you can use any programming language on top of Storm. We've added support for Ruby and Python, and support can easily be added for any language -- all you need to do is code a ~100 line library which implements a simple communication protocol with Storm.

3. Fault-tolerant: To launch a processing topology on Storm, all you have to do is provide a jar containing all your code. Storm then distributes that jar, assigns workers across the cluster to execute the topology, monitors the topology, and automatically reassigns workers that go down.

4. Horizontally scalable: All computations are done in parallel. To scale a realtime computation, all you have to do is add more machines and Storm takes care of the rest.

5. Reliable: Storm guarantees that each message will be fully processed at least once. Messages will be processed exactly once as long as there are no errors.

6. Fast: Storm is built with speed in mind. ZeroMQ is used for the underlying message passing, and care has been taken so that messages are processed extremely quickly.

Use cases for Storm

There are three broad use cases for Storm:

1. Stream processing: This is the traditional realtime processing use case: process messages and update a variety of databases.

2. Continuous computation: Storm can be used to do a continuous computation and stream out the results as they're computed. For example, we used Storm the other day to compute trending users on Twitter off of the Twitter firehose. Every second, Storm streams out the 50 users with the most retweets in the last few minutes with perfect accuracy. We stream this information directly into a webpage which visualizes and animates the trending users in realtime.

3. Distributed RPC: Distributed RPC is perhaps the most unexpected and most compelling use case for Storm. There are a lot of queries that are both hard to precompute and too intense to compute on the fly on a single machine. Traditionally, you have to do an approximation of some sort to lower the cost of a query like this. Storm gives the capability to parallelize an intense query so that you can compute it in realtime.

An example of a query that is only possible with distributed RPC is "reach": computing the number of unique people exposed to a URL on Twitter. To compute reach, you need to get all the people who tweeted the URL, get all the followers of all those people, unique that set of followers, and then count the number of uniques. It's an intense computation that potentially involves thousands of database calls and tens of millions of follower records. It can take minutes or worse to compute on a single machine. With Storm, you can do every step of the reach computation in parallel and compute reach for any URL in seconds (and less than a second for most URLs).

The idea behind distributed RPC is that you run a processing topology on Storm that implements the RPC function and waits for RPC invocations. An RPC invocation is a message containing the parameters of the RPC request and information of where Storm should send the results. The topology picks up messages, computes the RPC call in parallel, and returns the results to the return address.

Summary

Storm is already enormously useful for us at BackType. It reduces a ton of complexity in our realtime processing and lets us do things we couldn't do before. We look forward to the day we open source it.

You should follow the BackType tech team on Twitter here.

Permalink | Leave a comment  »

Source: http://tech.backtype.com/preview-of-storm-the-hadoop-of-realtime-proce

computer repair kansas city north computer repair kansas city missouri dell computer repair kansas city computer repair kansas city kansas

Worldwide Social Media users Top 1 Billion

According to the International Telecommunication Union, there are now over 1 billion people in the world who actively use social media. Of course, it will come as no surprise that Facebook tops the chart as the world?s most popular social network with over 900 million active users (Though differing opinions on what constitutes ?active? could [...]

Source: http://feedproxy.google.com/~r/TheGeekTechBlog/~3/RkEv-513Mh0/

computer repair kansas city reviews computer repair kansas city area computer repair kansas city overland park computer repair kansas city mo

Turn your iPhone into a two-player games console with GameDock, now seeking Kickstarter funding

GameDock is the latest innovative project to arrive on Kickstarter and it looks like a really interesting proposition. Aimed squarely at the retro gamers amongst us, it looks to provide an interface between your iPhone, iPad or iPod touch and your HDTV as well as giving you two handheld controllers to improve any gaming experience.

Source: http://feedproxy.google.com/~r/TheIphoneBlog/~3/DASzC3tjIrc/story01.htm

computer repair overland park ks computer repair shawnee ks computer repair services computer repair technician

Memorial Day 2012 holiday hours

Our business offices will be closed on Monday, May 28 to observe the US legal holiday. As always, we?ll provide same-day support for time-sensitive issues via our ticket and e-mail systems. However, questions that aren?t time-sensitive (including most billing matters) may not be answered until the next day, and telephone support (via callbacks) will be [...]

Source: http://blog.tigertech.net/posts/memorial-day-2012-holiday-hours/

computer repair kansas city ks computer repair kansas city north computer repair kansas city missouri dell computer repair kansas city

Spammers Have Started Using Android Botnets, Researchers Say

A new wave of pharmacy, penny stock and e-card spam emails are being sent by an Android botnet, according to security researchers from Microsoft and antivirus...

Source: http://feeds.pcworld.com/click.phdo?i=39d2771e5fbaa88ac448202e44d21ebd

computer repair kansas city mo computer repair kansas city ks computer repair kansas city north computer repair kansas city missouri

Arduino and LEDs Help You Give Everyone the Look of Disapproval

Want to really give someone a good glare? Let this Arduino project show you how to make a variety of scathing emoticons from LEDs.

Source: http://feeds.pcworld.com/click.phdo?i=646bf3d04a69540422567fe1af013a5a

computer repair kansas city 64151 computer repair kansas city reviews computer repair kansas city area computer repair kansas city overland park

Engine Yard explains how the PaaS market is evolving (infographic)

The market for platform-as-a-service (PaaS) companies will grow steadily in the next several years because of the potential for operational improvement and cost reduction, according to a just-released survey by Engine Yard.

Engine Yard competes heavily with Salesforce’s Force.com and …

Source: http://feeds.venturebeat.com/~r/Venturebeat/~3/cZUIPkXD2Hc/

computer repair kansas city area computer repair kansas city overland park computer repair kansas city mo computer repair kansas city ks

Extracting colors with Colorific

Source:

computer repair kansas city ks computer repair kansas city 64151 computer repair kansas city reviews computer repair kansas city area

Introducing ElephantDB: a distributed database specialized in exporting data from Hadoop

ElephantDB is a database that specializes in exporting key/value data from Hadoop. We have been running it in production at BackType for over half a year now and are excited to be open-sourcing it. In this post, I'll introduce ElephantDB, show how to use it, and then compare it to other databases out there. ElephantDB is hosted on GitHub here.

Unlike most other databases, ElephantDB dissassociates the creation of a database index from the serving of that index. ElephantDB is comprised of two components. The first is a library that is used in a MapReduce job to create an indexed key/value dataset that is stored on a distributed filesystem. The second component, ElephantDB server, is a daemon that downloads a subset of a dataset and serves it in a read-only, random-access fashion. A group of ElephantDB servers working together to serve a full dataset is called a ring. Both the creation and serving of a dataset are done in a fully distributed fashion.

99.9% of the complexity of distributed databases comes from supporting random writes. Since ElephantDB doesn't support random writes, it is extremely simple -- only a few thousand lines of code. Once a server loads up its subset of data, it does very little. This leads to ElephantDB being rock-solid in production, since there's almost no moving parts.

Why did we create ElephantDB?

We rely heavily on batch computation via Hadoop at BackType. We have over 25TB of social data (tweets, blog comments, etc.) stored on a distributed filesystem, and we use Hadoop to compute views of that data that we serve in our product. For example, one view is influence scores for everyone on Twitter. Another view is tweet counts for each url. We needed a database that could serve these various views to our product.

We looked into using Voldemort, but it lacked the incremental capabilities we wanted. We then realized that a database that doesn't support random writes is really simple and would be easy to write ourselves. We also thought that a database focused on only handling the batch exporting workflow could be made extremely simple to configure and operate. ElephantDB only took a few weeks to create and get into production.

High level overview

Before we dive into ElephantDB, let's get some terminology out of the way. An ElephantDB "domain" is analagous to a "table" from relational databases -- a single, related set of data. An ElephantDB domain is comprised of a fixed number of "shards" of a "Local Persistence". A "shard" is a subset of a domain, and a "Local Persistence" is a regular key/value database that reads and writes to disk. ElephantDB comes bundled with a "Local Persistence" implementation for Berkeley DB Java Edition. An ElephantDB "ring" is a group of servers that work together to serve a set of ElephantDB domains. Each server serves a subset of the data, and each server knows what data the other servers have. 

The following picture illustrates how an ElephantDB domain is created. Key/value pairs are sharded and indexed in a MapReduce job and stored on a distributed filesystem:

To serve the data, you point an ElephantDB ring where the domain is stored on the distributed filesystem. Each ElephantDB server will then download a subset of the shards and serve them, as illustrated in the following picture:

As you can see, the distributed filesystem is used as a distribution point for an ElephantDB domain.

Creating a domain of data via MapReduce

Let's look at how ElephantDB works in more detail. Let's start with how ElephantDB creates or updates an ElephantDB domain.

A domain is created or updated via a MapReduce job and stored on the distributed filesystem. Here's what a domain looks like on the distributed filesytem:

There can be multiple versions of a domain. Any given version of a domain is immutable, and everytime a domain is updated, a new version of the domain is created. In this picture, each numeric directory is a different version of the domain. A domain also contains a metadata file, domain-spec.yaml, that indicates what Local Persistence this domain is comprised of and how many shards are in this domain. The contents of the domain-spec.yaml for this domain is:

1 2 3

--- local_persistence: elephantdb.persistence.JavaBerkDB num_shards: 32

If we dive into one of these versions, we see the following directory structure. Each of these folders is a shard for this version of the domain:

Diving into one of these shards, we can see the Berkeley DB files for the shard:

When given a set of key/value pairs to turn into an ElephantDB domain, ElephantDB performs the following steps in a MapReduce job to create a version for the domain:

1. Select a shard for each key using consistent hashing (take the hash of the key and mod it by the number of shards)
2. Group all the key/value pairs by their shard number
3. In the reduce task, open a local persistence locally and stream all the key/value pairs for the shard into it.
4. At end of reduce task, copy the files for the local persistence to the distributed filesystem.

At no point in the creation of a domain is an ElephantDB server involved. The creation of a domain happens independently from serving the domain.

The code to create an ElephantDB domain is really simple. Here's an example using Cascading:

1 2 3 4 5 6

Tap source = new Hfs(new SequenceFile(new Fields("key", "value")), "/tmp/key-value-pairs"); DomainSpec spec = new DomainSpec(new JavaBerkDB(), 32); ElephantDBTap sink = new ElephantDBTap("/data/output/my-edb-domain", spec); Pipe p = new Pipe("pipe"); p = new ElephantTailAssembly(p, sink); new FlowConnector().connect(source, sink, p).complete();

First, we create an ElephantDBTap and configure it with how many shards we want and what LocalPersistence to use. At the end of our Cascading flow, we pipe the data through an ElephantTailAssembly which shards and groups the data to prepare it for the output tap.

ElephantDB is even easier to use with Cascalog:

1 2 3

(?-   (elephant-tap "/data/output/my-edb-domain" {:num-shards 32 :persistence-factory (JavaBerkDB.)} {})   (name-vars (hfs-seqfile "/tmp/key-value-pairs") ["?key" "?value"]))

"elephant-tap" from elephantdb-cascalog abstracts away the sharding/grouping process completely so that you can write key/value pairs to it just like you would write them to a regular file-based tap.  

Serving ElephantDB domains with ElephantDB server

Now let's take a look at how ElephantDB server works. A group of ElephantDB servers work together to serve a set of domains. Each ElephantDB server will serve a subset of each domain. The first thing you have to do is configure what domains the ring should be serving. The configuration is stored on the distributed filesystem and looks something like this:

1 2 3 4 5 6 7 8

{ :replication 1   :hosts ["edb1.mycompany.com" "edb2.mycompany.com" "edb3.mycompany.com"]   :port 3578   :domains {"tweet-counts" "/data/output/tweet-counts-edb"             "influenced-by" "/data/output/influenced-by-edb"             "influencer-of" "/data/output/influencer-of-edb"             } }

This configuration is as simple as it gets. You just have to list all the hosts in the ring, the replication factor, what domains to serve, and what port each server should use.

ElephantDB server decides what shards each host should serve by running a deterministic algorithm that assigns hosts to shards. Since every ElephantDB server gets the same input, they all make the same conclusion and know where all the data is located across the ring.

Each ElephantDB server also has a simple local configuration containing where to cache shards locally as well as machine-specific configuration for the local persistences. See the example conf in the project for an example of this.

ElephantDB server has a Thrift interface, so you can access it from any language. The interface contains methods for doing gets, multi-gets, and getting the status as to whether domains have been loaded from the distributed filesystem yet. Here's an example of doing a get to ElephantDB from Clojure:

1 2	(with-elephant-connection "edb1.mycompany.com" 3578 handler   (.getString handler "tweet-counts" "http://backtype.com"))

When you do a get to an ElephantDB server, it will route your request to the server that has the data for that key. This happens transparently.

When ElephantDB starts up, it asynchronously downloads shards for each domain from the distributed filesystem locally. You can use the status methods on its Thrift interface to know when it's fully loaded.

Currently, updating an ElephantDB server with new data requires taking downtime on the server. We are working on adding hot-swapping so that the server does this automatically in the background without taking downtime. 

Additional Features

Incremental updates

ElephantDB supports doing incremental updates of a domain. The steps ElephantDB performs to do an incremental update are just a little bit different than the steps ElephantDB uses to create a brand new domain:

1. Select a shard for each key using consistent hashing (just like before)
2. Group all the key/value pairs by their shard number (just like before)
3. Download the most recent version of the shard from the distributed filesystem
4. Execute updating logic for each new key/value pair
5. Upload the updated shard to the distributed filesystem as a new version

The updating logic is completely pluggable. By default, it just does a replace of whatever key/value pair was already in the local persistence. At BackType, we do all sorts of custom logic in the updater, from incrementing counts to merges of the old value with the new value.

All the incrementalization happens on the MapReduce side. Incrementally updating shards in this manner is a dramatic improvement over re-indexing the entire domain from scratch. 

Note that ElephantDB server updates itself by downloading all the new shards from scratch -- it's not able to download only the "new stuff". This typically isn't so bad as copying is quite fast, but it's certainly an area for improvement.

ElephantDB as an input source

Those ElephantDB domains sitting on the distributed filesystem can be used as an input source to your jobs. This is really useful. This lets you do analytics on your data without touching the servers that are serving the data to your application.

Furthermore, with this feature ElephantDB is useful even without ElephantDB server. ElephantDB can be used as an indexed key/value file format on top of Hadoop. There are lots and lots of applications for this.

Here's an example of using ElephantDB as an input source with Cascading. As you can see, the same tap that's used to create ElephantDB domains can be used as a source. The tap will emit key/value pairs into the processing flow.

1 2 3 4

ElephantDBTap source = new ElephantDBTap("/data/output/my-edb-domain"); Pipe p = new Pipe("pipe"); p = new Each(p, new Fields("key", "value"), new ProcessKeyValuePairs(), Fields.RESULTS); ...

Comparison to other database technologies

There are a lot of distributed databases out there. Most of these databases, like Cassandra and Riak, aren't comparable as they are read/write databases that create the index in the same place they serve the index. The most comparable database is Voldemort, which has a read-only mode with similarities to ElephantDB (Voldemort can also be used as a Dynamo-style read/write distributed database). Here's a tutorial from LinkedIn on how to use Voldemort's exporting features. Below is a comparison of Voldemort's read-only mode to ElephantDB. I'm sure I'm missing important aspects of Voldemort below. If so, let me know and I'll update the comparison.

Voldemort advantages:

1. Better performance: Voldemort has had a lot more optimization work done to it than ElephantDB, so it will have better performance. This is an area where ElephantDB may "borrow" from Voldemort (in particular, we'd like to try out Krati as an alternative to Berkeley DB JE).
2. Supports hot swapping: Voldemort databases can do live swaps of new versions of a domain in the background. This is functionality we're currently adding to ElephantDB so this difference will not last long.

ElephantDB advantages:

1. Simpler to configure: Since ElephantDB is a more specialized database, it benefits from being easier to configure. Whereas with Voldemort you need to manually assign hosts to partitions, ElephantDB just figures out the hosts to shard mapping on its own.
2. Support for incremental updates to a domain.
3. Support for reading ElephantDB domains stored on the distributed filesystem from MapReduce.
4. Very small codebase: ElephantDB is only a few thousand lines of code. This makes it easy to understand and extend.

The future

ElephantDB has been a very successful project at BackType. Future improvements we plan or would like to make to ElephantDB include the following:

1. Performance optimization: In particular, we'd like to try out Krati as an alternative to Berkeley DB JE.
2. Hot swapping: Currently, to update a domain without taking downtime, you'd need to do a rolling update of multiple ElephantDB rings. To simplify this workflow, we'd instead like for a live ElephantDB ring to download new shards and swap them in the background.
3. Richer data model: We'd like to explore adding a richer data model to ElephantDB beyond key/value.

Otherwise, we want to keep ElephantDB simple. We believe that a simple database is more reliable and makes life easier for everyone.

Conclusion

We spend almost no time on operational or maintenance issues with ElephantDB at BackType. It just works. As a small startup, it is important for us to eliminate complexity from our systems so that we can focus on customer problems. ElephantDB removes all the friction from serving batch-computed views of our 25TB dataset into our application. Its incremental capabilities let us keep those views updated with minimal cost, and it wears multiple hats by letting us use an ElephantDB domain as an input source to our jobs.

If you're interested in using ElephantDB, be sure to join the user group and follow the project on GitHub.

You should follow the BackType tech team on Twitter here.

 

Permalink | Leave a comment  »

Source: http://tech.backtype.com/introducing-elephantdb-a-distributed-database

computer repair kansas city mo computer repair kansas city ks computer repair kansas city north computer repair kansas city missouri