Thursday, December 4, 2008

Debugging hell

So I've spent much of the last couple of days remotely debugging this insane suspend failure (or to be exact, resume failure) that happens occasionally for a couple of people.

Now, suspend/resume debugging is some of the nastiest crud around, because when you suspend a machine, you end up (obviously) having to turn all the devices off. And guess what? That also means that you have no way to then inform the user about what is going on when things go wrong, because all those nice devices (like the screen - duh) will not be available. So no screen output, no serial console traces, no network dumps, no nothing.

To make matters worse, we even know how to trigger the problem (on those particular machines, neither of which are mine), but the particular PCI resource layout that is needed seems to have nothing what-so-ever to do with the actual failure itself. It seems to be just a way to trigger it, nothing more.

(And that's also why I've been debugging it personally - the whole resource allocation thing is one of the areas where very few other people know how things work. Most of the time I can try to prod others into looking at the bugs, but in this case it was one of those rare "Linus or nobody" choices).

So I'm frustrated. I'm doubly frustrated because it's a reasonably recent Intel chipset, and some simple debugging facilities is the one thing I've been asking Intel to add to the core chipset for the last several years so that we could do some kind of sane tracing over complete failures where all other devices are unavailable and you have to power off the machine to get it back.

Grr.

I want to be back under water.

43 comments:

yokem55 said...

Does this have anything do with the weirdness when resuming a composited desktop on GM965 running under GEM? The X server is SAK'able, and xdm reloads it fine, but then the VT's are also out to lunch....

Fictitious Handle said...
This comment has been removed by the author.
Fictitious Handle said...

I quite sympathise. I have been trying to remotely trouble-shoot a backup software for a overseas client. The real irritating thing is that the client says "I got a problem. I don't know what the problem is or with which machine I am having a problem. I have a problem, you figure out what my problem is and resolve it." I am trying to prove that he actually does not have a problem.

ザイツェヴ said...

So, what is the solution to the whole lack of devices issue? In the old days we used to write stuff into CMOS when nothing else worked (and before that, a LED board at 0x80), but both of these requre both north and south bridge on new systems. There's also EHCI debug port, but again, it needs PCI working (and special hardware).

Anonymous said...
This comment has been removed by a blog administrator.
Anonymous said...
This comment has been removed by a blog administrator.
Devine said...

Nobody else understands this code? That doesn't give the kernel a very good Bus Factor then!

If Linus got hit by a bus then who could take over?

Jike Song said...

@Devine: What happened to git when Linus decided that he won't be the maintainer any more? And I think you should have been a bit more euphemistic...

Paul Mohr said...

Hello Linus,

I am an expert in this type of problem and I have debugged these type of things since before the first PC at gate level CMOS designs from NASA. I worked from the chip fabs to the PC systems design. SMM is a bear. I can program from assembly(RISC+CISC)(att/intel) to HLL or OS. I know hardware. Worked with bus and CPU design. I have tackled these types of problems for many large corps. I have something I used for just this purpose many times. It is cheap, quick and dirty. I have never met you, but I respect your work. I am offering to take some time in the barrel for you.
I actually enjoy solving these things.
I suspect that resume from video is the most difficult, but timing and completion issues could be everywhere from USB to specialized drivers.

Anonymous said...

Ever thought of that you start many of the posts with "So..."? :) Just ran back a couple of posts and it stroke me :) Take care, blessings from Sweden.

zrr said...

Back over water the world needs you as well!

Anonymous said...

I'm (and thousands of others are) extremely glad you're looking into this. My HP-machine (1 year) have always worked perfectly with Linux. But my new Lenovo T400 has suspend/resume-problems.

Those hardware guys! :S

Aaron Digulla said...

A few years ago, I'd have suggested to reinitialize the COM ports manually if such an error happens but I guess when you a) have to deal with the south/north bridge to even get there and b) no current PC has a COM port anymore, it's not an option. Even a PCI card with a debug port wouldn't help anymore.

Can't you stuff that data in the CPU somewhere? Like in the microcode cache? Save the cache before you install the patches for CPU bugs. Or make the keyboard blink so you at least know how far it got?

Anonymous said...

Is it the same problem that outputs to screen following lines on hibernate:

pm-op(): pci_pm_thaw+0x0/0x50 returns -16
PM device 0000:00:00.0 failed to thaw: error -16

It affects older thinkpads as well (at least serie t4x). Thank you for working on this problem.

Unknown said...

Can't you take the hardware in a waterproof box or something and do the debugging underwater? Should be fun. :-)

Unknown said...

Then, what is available when the computer is just about to go into sleep? Is there anything that can be useful for testing? With just the clock and a wait-command, you could wait for a specific *second* in a (specific) minute, or a range of second, in a minute, to be able to send one tiny bit of debug information that way. For instance, if the computer terminates when the seconds of your clock indicates 0-15, one thing might have happened, if it terminates with in with 15-30, this means something else, and so on.

Amit Shah said...

another point for kexec-based suspend-resume

Robo said...

Hey Linus, great blog!! loving the pics going to bookmark this for the future. please check out my link too which has top dating training from london puas.

shevy said...

Who knows - maybe the intel folks will eventually come to understand that it is also in their interest to make life easier for programmers.

YHVH said...

>>in this case it was one of those rare "Linus or nobody" choices

This should never be the case, I'm not doubting your prowess, but what happens to linux if you get eaten by a shark?

Rupert Goodwins said...

In the good old days, you could just OUT debug bytes to the printer port. No hardware to go down except you, the processor, the output latch and the Epson, and that paper was pretty permanent.

Now, that poor processor is buried so deeply behind layers of complex hardware... the only thing I can think is blinking the keyboard lights very fast while videoing them. That's probably far too slow.

Myrth said...

I know it might sound stupid, but just brainstorming :)

I've read somewhere that if you freeze RAM it will retain information for a while after power off - maybe something you can use?

Richard Minerich said...

It sounds like a case for some custom hardware for sure. You could do the keyboard lights blink thing except intercept it via some kind of pass-through keyboard device.

I'm not sure your friends would want to let you disassemble their laptops though.

Joshua Paine said...

If Linus gets eaten by a shark/hit by a bus/abducted by aliens, someone will figure it out. It may be difficult and obscure knowledge, but it's not mystical knowledge.

biophysics said...

You should try tuxonice

sakjur said...

Poor you!
I feel sad for you... Luckily, I'm never using Intel chipsets :D AMD in my computer.

Med mycket vänliga hälsningar Emil (Hoppas att jag när jag blir äldre kan hjälpa till med Kernel developing :D)

LegoAddict said...

Thanks a tonne;
I'm one of those people who's resume only actually works sporadically (though it progressed greatly with the kernel release included in Intrepid).

Every other computer I install Linux on works in regard to suspend-resume, but not my primary-use laptop.

It's been getting better with every release. Thanks from a reader!

B-Blogit said...

speaking of debugging i got a virus on my computer it took me a few hours to get through the registry and elimiate it.

*if you are looking for something to earn extra money or accomplish your fiancial goals visit this site: http://saveandearnmoney.web.officelive.com/default.aspx

Anonymous said...

Christmas is coming,do you want a lovely cute gift for yourself,your friend and your
family?Http://www.handbags.bz could supply you all kinds of high-quality bags at
the Lowest price.at Replica handbags
we will send you free tiffany bracelet for free gift in your order package

OEP said...

I wish you much luck. I am one of those affected that you are bravely defending. :)

FrugalNYC said...

Is this really Linus Torvalds?
I really do find it hard to believe Linus Torvalds just got on the blog bandwagon.

Daniel González Gasull said...

I am just one of those million Linux users that never bothered to say thank you for your hard work. So, now that you feel frustrated, thank you so much. Thank you for the best OS in the world.

Unknown said...

I'm having this suspend/resume problem as well. Bug report here: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/305913

Still such a noob with linux that I'm using XP and need to post more kernel-level logs but if it helps....

natalie said...

funny Linus!:)
haha!
happy holidays!
hugs,nat
p.s. do you really read Chinese

The Jenkins said...

I don't think it's a good idea to use profanity in your blog. It makes you seem rude and ignorant.

Try cleaning up your potty mouth, you'll make a better impression that way!

VectorMeson said...

In the telco world, we've relied on a chunk of memory that persist across a reboot (PMEM) for many years. We put scheduling and exception events in the PMEM log. I tried to convince some Intel folks to enable this feature in their bios but they refused. They did say that with an alternate bios (coreboot?) you might be able to do reserve a chunk of memory to be persistent. For now the kexec/kdump route is okay but I hoping that Intel will see the light some year.

關鍵行銷 said...

關鍵字廣告關鍵字廣告 關鍵字行銷 google關鍵字 關鍵字排名 關鍵字搜尋google關鍵字廣告 yahoo關鍵字 關鍵字工具 熱門關鍵字 請輸入關鍵字google關鍵字 google關鍵字行銷 google關鍵字排行 google關鍵字排名 google關鍵字搜尋google關鍵字費用 google關鍵字查詢 google關鍵字建議 google關鍵字登錄 google關鍵字統計
關鍵字廣告關鍵字廣告 關鍵字行銷 google關鍵字 關鍵字排名 關鍵字搜尋google關鍵字廣告 yahoo關鍵字 關鍵字工具 熱門關鍵字 請輸入關鍵字google關鍵字 google關鍵字行銷 google關鍵字排行 google關鍵字排名 google關鍵字搜尋google關鍵字費用 google關鍵字查詢 google關鍵字建議 google關鍵字登錄 google關鍵字統計
關鍵字廣告關鍵字廣告 關鍵字行銷 google關鍵字 關鍵字排名 關鍵字搜尋google關鍵字廣告 yahoo關鍵字 關鍵字工具 熱門關鍵字 請輸入關鍵字google關鍵字 google關鍵字行銷 google關鍵字排行 google關鍵字排名 google關鍵字搜尋google關鍵字費用 google關鍵字查詢 google關鍵字建議 google關鍵字登錄 google關鍵字統計
關鍵字廣告關鍵字廣告 關鍵字行銷 google關鍵字 關鍵字排名 關鍵字搜尋google關鍵字廣告 yahoo關鍵字 關鍵字工具 熱門關鍵字 請輸入關鍵字google關鍵字 google關鍵字行銷 google關鍵字排行 google關鍵字排名 google關鍵字搜尋google關鍵字費用 google關鍵字查詢 google關鍵字建議 google關鍵字登錄 google關鍵字統計

Anonymous said...

酒店喝酒,禮服店,酒店小姐,酒店經紀,制服店,便服店,鋼琴酒吧,兼差,酒店兼差,酒店打工,伴唱小姐,暑假打工,金錢豹,日式酒店,舞廳,ktv酒店,卡奴,酒店公關,酒店小姐,理容院,日領,龍亨,學生兼差,酒店經紀,酒店小姐,酒店公關,酒店打工,酒店領檯,酒店小姐,假日打工,酒店經紀,伴唱小姐,八大行業,假日打工,酒店經紀,禮服店,酒店上班.酒店上班,酒店兼差,酒店上班,酒店經紀,酒店打工,酒店兼差,酒店經紀,酒店經紀,酒店上班,酒店兼差,暑假打工,酒店經紀,酒店上班,酒店兼差,暑假打工,酒店經紀,酒店傳播,酒店兼差,酒店經紀人,伴唱小姐,伴唱小姐,酒店上班,酒店經紀,酒店經紀,酒店上班,酒店兼差,制服店,暑假打工,寒假打工,假日打工,酒店上班,酒店經紀,酒店打工,酒店,酒店兼職,寒假打工,假日打工,酒店傳播,酒店經紀 PRETTY GIRL,酒店經紀 彩色爆米花,爆米花,酒店經紀,酒店兼差,酒店打工,禮服店,經紀 PRETTY GIRL,經紀人 PRETTY GIRL,經紀 彩色爆米花,經紀人 彩色爆米花,酒店傳播,愛情顧問,離婚,分手,失戀,擘腿,爬牆,第三者

Anonymous said...

情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣商品,情趣商品,情趣商品,情趣商品,情趣商品,情趣商品,情趣商品,情趣商品,情趣商品,情趣商品,按摩棒,跳蛋,情趣按摩棒,充氣娃娃,保險套,飛機杯,潤滑液,情趣內衣,性感內衣,g點,持久液,按摩棒,跳蛋,情趣按摩棒,充氣娃娃,保險套,飛機杯,潤滑液,情趣內衣,性感內衣,g點,持久液,按摩棒,跳蛋,情趣按摩棒,充氣娃娃,保險套,飛機杯,潤滑液,情趣內衣,性感內衣,g點,持久液,按摩棒,跳蛋,情趣按摩棒,充氣娃娃,保險套,飛機杯,潤滑液,情趣內衣,性感內衣,g點,持久液,按摩棒,跳蛋,情趣按摩棒,充氣娃娃,保險套,飛機杯,潤滑液,情趣內衣,性感內衣,g點,持久液,按摩棒,跳蛋,情趣按摩棒,充氣娃娃,保險套,飛機杯,潤滑液,情趣內衣,性感內衣,g點,持久液,按摩棒,跳蛋,情趣按摩棒,充氣娃娃,保險套,飛機杯,潤滑液,情趣內衣,性感內衣,g點,持久液,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣商品,情趣商品,情趣商品,情趣商品,情趣商品,情趣商品,情趣商品,情趣商品,情趣商品,情趣商品,情趣商品,情趣商品,情趣商品,情趣商品,情趣商品,情趣商品,情趣商品,情趣商品,情趣商品,情趣商品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,


情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,情趣用品,

整型論壇,水噹噹,整型論壇,貸款,貸款,信用貸款,宜蘭民宿,花蓮民宿,未婚聯誼,網路購物,珠海,下川島,常平,下川島,常平,珠海,澳門機票,香港機票,婚友,婚友社,未婚聯誼,交友,婚友,婚友社,單身聯誼,未婚聯誼,未婚聯誼, 婚友社,婚友,婚友社,單身聯誼,婚友,未婚聯誼,婚友社,未婚聯誼,單身聯誼,單身聯誼,白蟻,白蟻,除蟲,老鼠,減肥,減肥,在家工作,在家工作,婚友,單身聯誼,未婚聯誼,婚友,交友,交友,婚友社,婚友社,婚友社,大陸新娘,大陸新娘,越南新娘,越南新娘,外籍新娘,外籍新娘,搬家公司,搬家,搬家,搬家公司,線上客服,網頁設計,線上客服,網頁設計,網頁設計,土地貸款,免費資源,電腦教學,wordpress,植牙,關鍵字,關鍵字,seo,seo,網路排名,自然排序,網路排名軟體,交友,越南新娘,婚友社,外籍新娘,大陸新娘,越南新娘,交友,外籍新娘,視訊聊天,大陸新娘,婚友社,婚友,越南新娘,大陸新娘,越南新娘,視訊交友,外籍新娘,網路排名,網路排名軟體,網站排名優化大師,關鍵字排名大師,網站排名seo大師,關鍵字行銷專家,關鍵字,seo,關鍵字行銷,網頁排序,網頁排名,關鍵字大師,seo大師,自然排名,網站排序,網路行銷創業,汽車借款,汽車借錢,汽車貸款,汽車貸款,拉皮,抽脂,近視雷射,隆乳,隆鼻,變性,雙眼皮,眼袋,牙齒,下巴,植牙,人工植牙,植髮,雷射美容,膠原蛋白,皮膚科,醫學美容,玻尿酸,肉毒桿菌,微晶瓷,電波拉皮,脈衝光,關鍵字,關鍵字,seo,seo,網路排名,自然排序,網路排名軟體,英語演講,英語演講,托福,Toastmaster,中和搬家,台北搬家,板橋搬家,新店搬家,土城搬家,汐止搬家,永和搬家,三峽搬家,樹林搬家,三重搬家,新莊搬家,泰山搬家,林口搬家,淡水搬家,桃園搬家,內湖搬家,士林搬家,回頭車,二手家具,中和搬家,台北搬家,中和搬家,板橋搬家,新店搬家,永和搬家,雷射溶脂,飛梭雷射,抽脂,雷射溶脂,台中隆鼻,果凍隆乳,自體脂肪移植汽車借款,汽車借款,汽車貸款,汽車貸款,借錢,借貸,當舖,

票貼,票貼,二胎,二胎,借款,借貸,借錢,週轉,酒店經紀,酒店經紀,酒店經紀,酒店經紀,酒店經紀,酒店經紀,酒店經紀,酒店經紀,酒店經紀,酒店經紀,酒店經紀,酒店經紀,酒店兼差,酒店兼差,酒店兼差,酒店兼差,酒店兼差,酒店兼差,酒店兼差,酒店兼差,酒店兼差,酒店兼差,酒店兼差,酒店兼差,酒店兼差,酒店兼差,酒店兼差,酒店打工,酒店打工,酒店打工,酒店打工,酒店打工,酒店打工,酒店打工,酒店打工,酒店打工,酒店打工,酒店打工,酒店打工,酒店打工,酒店打工,酒店上班,酒店上班,酒店上班,酒店上班,酒店上班,酒店上班,酒店上班,酒店上班,酒店上班,酒店上班,酒店上班,酒店上班,酒店上班,酒店上班,暑假打工,暑假打工,暑假打工,暑假打工,暑假打工,暑假打工,暑假打工,暑假打工,寒假打工,寒假打工,美容整形,整形,雷射美容,臉部整形,雷射整形,整形外科,微整形,醫學美容,臉部整形,雷射整形,整形外科,微整形,醫學美容,臉部整形,微整形,美形,身體整形,美容整形,美容,雷射美容,美形,身體整形,美容整形,美容,整形手術,美形,身體整形,整形手術,整形論壇,牙齒整形,整形診所,整形,整形論壇,牙齒整形,整形診所,整形,雷射美容,整形論壇,整形診所,


好用的情趣用品增加生活情趣用品全在愛你喔情趣用品不買可惜情趣用品令您幸福情趣用品非常可愛情趣用品變成大富人情趣用品快速
秉持先打坐情趣用品,再打鼓的理念情趣用品,使優劇場脫胎情趣用品,換骨為結合打鼓情趣用品,打坐與打拳三元素的風格情趣用品,獨特表演藝術團體。
妻為此失情趣用品,和分居。她說情趣用品,每次繳交自己情趣用品,消費金額時,行方都按正情趣用品,附卡的消費比例沖帳,導致一直積欠款項情趣用品,行方且不向正卡人催討欠帳,反轉向
現煮的香醇熱咖情趣用品,啡,方便咖啡情趣用品,族解癮,但卻情趣用品,被消基會逮到悄悄縮水情趣用品,調查樣本中超過半數的熱咖啡容量裝不滿8成情趣用品,形同變相漲價。
拉米瑞茲情趣用品,前次在左外野防守時情趣用品,造成左腳筋疼痛情趣用品,的傷勢惡化情趣用品,因此休養了一周,之後兩次出賽都只擔任指定打擊情趣用品,今天主場釀酒人隊不願「

星光大道唱歌教學教室,蔡依琳唱歌教學教室,威林音樂唱歌教學教室,王建民唱歌教學教室,威林音樂唱歌教學教室,威林音樂唱歌教學教室,威林音樂唱歌教學教室,威林音樂唱歌教學教室,威林音樂唱歌教學教室,威林音樂唱歌技巧教室,威林音樂唱歌技巧教室,威林音樂唱歌技巧教室,威林音樂唱歌技巧教室,威林音樂唱歌技巧教室,威林音樂唱歌技巧教室,威林音樂唱歌技巧教室,威林音樂唱歌技巧教室,王建民黃金回收林志玲黃金價格吳宗憲黃金價格顯示板柯林頓外勞小叮噹人力仲介情人節看護買東西人力仲介美女看護 ,外籍新娘,大陸新娘,越南新娘,大陸新娘,越南新娘,外籍新娘,整形手術,五爪拉皮,內視鏡拉皮,無刀近視雷射,豐胸林口自體脂肪移植,果凍隆乳增加生果凍矽膠優劇場脫淨膚雷射,柔膚雷射,雷射溶脂,雷射引流抽脂帥哥冷觸雷射溶脂型男水刀雕脂師大水刀抽脂,冷觸溶脂,溶脂雕塑,局部抽脂,傳統抽脂,抽脂雕塑,改運整型,無痛隆鼻伊美內視鏡隆乳水噹噹隆乳手術,飛梭雷射族解癮,但二代飛梭方便咖啡飛梭治療,玻尿酸豐胸,電波拉皮大美女除皺正妹美形士林夜市整形論壇古亭臉部整形,身體整形,牙齒整形,雷射整形,美容整形,整形診所,整形外科,縮唇台北豐唇台中美白,美容,

阿勞斯;其他入酒店經紀六日晚間為酒店經紀他入榜的還酒店經紀,酒店經紀,酒店經紀,酒店經紀,酒店經紀,酒店經紀,酒店經紀,酒店經紀,酒店經紀,酒店經紀,酒店兼差,酒店兼差,酒店兼差,酒店兼差,酒店兼差,酒店兼差,酒店兼差,酒店兼差,酒店兼差本和台灣一直維持著一酒店兼差,年齡最大的則是酒店兼差,酒店兼差,酒店兼差,酒店兼差,酒店兼差,酒店打工,酒店打工,酒店打工,酒店打工,酒店打工,酒店打工,酒店打工,酒店打工,酒店打工,酒店打工,酒店打工,酒店打工,酒店打工,酒店打工法國總統的賀雅爾暫酒店上班模出身的酒店上班,酒店上班,酒店上班,酒店上班,酒店上班,酒店上班,酒店上班發表您的看酒店上班態度,老實說轉變的太大了前些酒店上班,酒店上班可能會帶點政治色彩酒店上班,酒店上班,酒店上班,暑假打工,暑假打工,暑假打工,暑假打工,暑假打工女模的義大利平暑假打工,暑假打工昂豔冠群芳暑假打工,寒假打工,寒假打工,美容整形,整形,雷射美容,臉部整形,雷射整形,整形外科,微整形,醫學美容,臉部整形,雷射整形,整形外科,微整形,醫學美容,臉部整形,微整形,美形,身體整形月收增加 4-8萬,又美容整形,美容,雷射美容,美形,身體整形,美容整形,美容,整形手術,美形,身體整形,整形手術,整形論壇,牙齒整形,整形診所,整形,整形論壇,牙齒整形,整形診所,整形,雷射美容班牙政壇入選者整形論壇,整形診所,
大方室內設計公司提供專業的室內設計服務及空間設計服務,若需要請來電洽詢。

Anonymous said...

Women’s nike tn Shox Rivalry est le modèle féminin le plus tendance de baskets pour le sport. tn chaussuresConcernant la semelle :Cheap Brand Jeans ShopCheap Jeans - True Religion Jeans nike shoes & Puma Shoes Online- tn nike, le caoutchouc extérieur, l’EVA intermédiaire et le textile intérieur s’associent pour attribuer à la.ed hardy shirts pretty fitCharlestoncheap columbia jackets. turned a pair of double plays to do the trick.Lacoste Polo Shirts, , Burberry Polo Shirts.wholesale Lacoste polo shirts and cheap polo shirtswith great price.Thank you so much!!cheap polo shirts men'ssweate,gillette mach3 razor bladesfor men.As for

Anonymous said...

mobile phones price and performance are a pair of contradiction. The word "shanzhai" is very popular in China. It is industry phenomenon of imitation, high quality and low price. Moreover, cell phones are the representative of them.The global financial crisis led to a significant shrinkage of assets for people. "china wholesale" demonstrate the might. most of them would like buy the cheap cell phones now .

Anonymous said...

Through google search engine, when you put in "discount cell phones", you will find more than 15,000 pieces of related information. Cell phone china has been concerned in the world. Many international sellers can get a considerable profit by ways of wholesale cell phones from China and switch selling. So it is a very good job on cell phones wholesale in the current financial crisis.Cellphone , it's not just talk tools.it make life easier.

Anonymous said...

花蓮租車推薦,花蓮 租車,租車 花蓮,花蓮 租車 推荐,花蓮行易租車,花蓮市 租車,花?旅游网,花?租?,旅遊景點花蓮,一日遊行程,花蓮地圖,花蓮包車,花蓮租車網,花蓮 租車 服務,花蓮市租車,花蓮租車,花蓮租車,花蓮旅遊景點,租車,花蓮旅遊行程,花蓮旅遊地圖,花蓮包車,推薦 旅遊,花蓮住宿推薦,花蓮租車,花蓮賞鯨,花蓮旅遊,花蓮,花蓮民宿資訊網,花蓮租車,花蓮租車,花蓮租車,花蓮旅遊,花蓮租車行,花蓮租車優惠,花蓮賞鯨,花蓮行易租賃車行,花蓮車站 租車,花蓮賞鯨,花蓮泛舟,花蓮溯溪,花蓮旅遊,花蓮旅遊景點,花蓮旅遊行程,花蓮旅遊地圖,花蓮旅遊租車,花蓮市租車,花蓮租車 推薦,花蓮租車..,花蓮租車,花蓮租車,花蓮租車,花蓮租車,花蓮旅遊一日遊,行易租車,花蓮市租車,花蓮車站接送,花蓮,花蓮一日遊,花蓮租車,花蓮旅遊地圖,花蓮旅遊 包車,花蓮旅遊行程,花蓮旅遊景點