Wednesday, 11 September 2013

multiple threads using same value fetched by database DAO class method

multiple threads using same value fetched by database DAO class method

Status: unsolved mistery
This is really deep and n*eeds patience* to help. Help is highly
regarded!!!!!!
I know following would be a mess but I couldn't find a better description
to ask. apologies for that.
I had to make a pastebin as I had to point out line numbers.
main app, http://pastebin.com/i9rVyari logs, http://pastebin.com/2c4pU1K8
I am starting many threads (10) in a for loop with instantiating a
runnable class but it seems I am getting same result from db (I am geting
some string from db, then changing it) but with each thread, I get same
string (despite each thread changed it.) . using jdbc for postgresql what
might be the usual issues ?
line 252
and line 223
the link is marked as processed. (true) in db. other threads of crawler
class also do it. so when line 252 should get a link. it should be
processed = false. but I see all threads take same link.
when one of the threads crawled the link . it makes it processed = true.
the others then should not crawl it. (get it) is its marked processed =
true.



getNonProcessedLinkFromDB() returns a non processed link
public String getNonProcessedLink(){ 645
public boolean markLinkAsProcesed(String link){ 705
getNonProcessedLinkFromDB will see for processed = false links and give
one out of them . limit 1 each thread has a starting interval gap of 20
secs.
within one thread. 1 or 2 seconds (estimate processing time for crawling)
line 98 keepS threads from grabbing the same url
if you see the result. one thread made it true. still others access it.
waaaay after some time.
all thread are seperate. even one races. the db makes the link true at the
moment the first thread processes it

No comments:

Post a Comment