ruby multi process with fork
Why I didn't use thread
I use ruby fork function for Multi process
First time I used multiple thread but multiple thread is only run on single core
only single core usage is 100%.
another core is not used by multiple thread.
because MRI has GIT (Global Interpreter Lock)
First time I used multiple thread but multiple thread is only run on single core
only single core usage is 100%.
another core is not used by multiple thread.
because MRI has GIT (Global Interpreter Lock)
I did use mutiple process with fork
I change multiple thread to multiple process. that's why I want save time and using all multiple core
And it's well working on multiple core.
my laptop has physical 4 core. multiple process used all core usage 100% and total usage is 400%(4core)
this source for http image download and get image width, height
and print console if image has not specific width, height
And it's well working on multiple core.
my laptop has physical 4 core. multiple process used all core usage 100% and total usage is 400%(4core)
this source for http image download and get image width, height
and print console if image has not specific width, height
require 'mini_magick' require 'csv' def image_size_check(datas) i = 0 num = datas.length while i < num do data = datas[i] begin url = data[:thumbnailurl].strip image = MiniMagick::Image.open(url) width = image[:width] height = image[:height] puts "#{i},#{width},#{height},#{data[:pageid]},#{data[:menuid]},#{data[:itemid]},#{data[:namekr]},#{data[:thumbnailurl]}" unless (width == 372 and height == 209) or (width == 289 or height == 419) rescue StandardError => e puts "#{i},#{width},#{height},#{data[:pageid]},#{data[:menuid]},#{data[:itemid]},#{data[:namekr]},#{data[:thumbnailurl]},exception" end i +=1 end end data = Array.new CSV.foreach("item_thumbnail.csv", { encoding: "UTF-8", headers: true, header_converters: :symbol, converters: :all}) do |row| data << row.to_hash end data.each_slice(3000) {|a| Process.fork do image_size_check(a) end } Process.wait
Run
$ run image_size_check.rb >> out
Monitoring
$ tail -f out